From 01dc99ca4479c318a37fbe3b318eaae5a9419d02 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Mon, 22 Jun 2020 11:40:34 -0500 Subject: [PATCH 001/140] Add RFC for App Mixins Signed-off-by: Joe Kutner --- text/0000-app-mixins.md | 187 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 187 insertions(+) create mode 100644 text/0000-app-mixins.md diff --git a/text/0000-app-mixins.md b/text/0000-app-mixins.md new file mode 100644 index 000000000..0115b6c9b --- /dev/null +++ b/text/0000-app-mixins.md @@ -0,0 +1,187 @@ +# Meta +[meta]: #meta +- Name: Application Mixins +- Start Date: 2020-06-22 +- Author(s): [Joe Kutner](https://github.com/jkutner/) +- RFC Pull Request: (leave blank) +- CNB Pull Request: (leave blank) +- CNB Issue: (leave blank) +- Supersedes: N/A + +# Summary +[summary]: #summary + +This is a proposal for allowing application developers (buildpack users) to specify mixins that will be dynamically included with their build and run images. + +# Motivation +[motivation]: #motivation + +Mixins already allow buildpack authors to create buildpacks that depend on an extended set of OS packages without affecting build time, but it's common for application code to depend on OS packages too. But allowing buildpacks to arbitrarily install OS packages during build would drastically increase build time, especially when CNB tooling is used to build or rebase many apps with similar package requirements. For that reason, this proposal defines a mechanism to implement mixin support for application developers without significant impact on build performance. + +# What it is +[what-it-is]: #what-it-is + +- *application developer* - a person who is using buildpacks to transform their application code into an OCI image +- *stackpack* - a new type of buildpack that runs against the stack image(s) instead of an app + +A application developer may specify a list of mixins in their application's `project.toml` file like this: + +```toml +[build] +mixins = [ "" ] +``` + +When a command like `pack build` is run, the list of mixins will be processed before buildpacks are run. For each mixin name, the following will happen: + +* If the mixin name is prefixed with `build:` (as per [RFC-0006](https://github.com/buildpacks/rfcs/blob/main/text/0006-stage-specific-mixins.md)) the mixin will be dynamically added to the build image. +* If the mixin name is prefixed with `run:` (as per [RFC-0006](https://github.com/buildpacks/rfcs/blob/main/text/0006-stage-specific-mixins.md)) the mixin will be dynamically added to the run image. +* If the mixin name does not have a prefix it will be dynamically added to both the build and run stack images. + +# How it Works +[how-it-works]: #how-it-works + +When a list of mixins are defined in a `project.toml` and `pack build` is run against that project, the following will happen: + +1. The lifecycle will compare the list of mixins to those provided by the stack. If all mixin names are provided by the stack, no further action is required. +1. In any requested mixin is not provided by the stack, the lifecycle will run the detect phase for all stackpacks defined in the builder. +1. The lifecycle will compare the mixins added to the build plan to see if they match they required mixins. +1. If at least one stackpack passes detect and provides the required mixin(s), the lifecycle will execute the stackpack build phase for passing stackpack(s). If no stackpacks pass detect, or no stackpacks provide the required mixin, the build will fail with a message that describes the mixins that could not be provided. +1. During the lifecycle's build phase, the stackpacks that passed detection will run against the build and run images accordingly (see details below). All stackpacks will be run before the user's buildpacks. + +## Stackpacks + + A stackpacks is a special case of buildpack that has the following properties: + +* Is run as the `root` user +* Configured with `privileged = true` in the `buildpack.toml` +* Can only create one layer +* Does not have access to the application source code +* Is run before all regular buildpacks +* Is run against both the build and run images +* Is distributed with the builder image +* May not write to the `/layers` directory + +The stackpack interface is identical to the buildpack interface (i.e. the same `bin/detect` and `bin/build` scripts are required). However, some of the context it is run in is different from regular buildpacks. + +For each stackpack, the lifecycle will use [snapshotting](https://github.com/GoogleContainerTools/kaniko/blob/master/docs/designdoc.md#snapshotting-snapshotting) to capture changes made during the stackpack's build phase (excluding `/tmp`, `/cnb`, and `/layers`). Alternatively, a platform may store a new stack image to cache the changes. All of the captured changes will be included in a single layer produced as output from the stackpack. The `/layers` dir MAY NOT be used to create arbitrary layers. + +A stackpack is included in a builder by defining it in the `builder.toml` like any other buildpack: + +``` +[[buildpacks]] +id = "" +uri = "" +``` + +## Example: Apt Buildpack + + A buildpack that can install an arbitrary list of mixins would have a `buildpack.toml` like this: + + ```toml +[buildpack] +id = "example/apt" +privileged = true +``` + +Its `bin/detect` would have the following contents: + + ```bash +#!/usr/bin/env bash + +cat <"$2" +[[mixins]] +type = "package" +name = "*" +TOML +``` + +Its `bin/build` would have the following contents: + + ```bash +#!/usr/bin/env bash + +apt update -y + +for package in $(cat $3 | yj -t | jq -r ".entries | .[] | .name"); do + apt install $package +done + +rm -rf /var/cache/apt/archives +``` + +## Upgrading an App + +When attempting to rebase an image that used a stackpack to provide a mixin, the platform will attempt to re-run the stackpack(s) during rebase. + +When the base image has not changed +1. If the buildpacks are configure as idempotent (the default) load the previous layers onto the base images. +1. Run the buildpacks, creating an ephemeral image. +1. Rebase the app on to the new ephemeral image. + +When there is an update to either the build or run base images +1. Pull the new base image(s) +1. Run the root buildpacks against the new image(s) without loading previous layers, and create an ephemeral image. +1. Rebase the app on to the new ephemeral image. + +**Question**: should some buildpacks be able to opt-out of this rebase behavior (i.e. they are able to guarantee ABI compatibility)? + +# Drawbacks +[drawbacks]: #drawbacks + +- Mixins are less flexible that a generic [root buildpack](https://github.com/buildpacks/rfcs/blob/app-image-ext-buildpacks/text/0000-app-image-extensions.md). + +# Alternatives +[alternatives]: #alternatives + +- [Add root buildpack as interface for app image extension RFC](https://github.com/buildpacks/rfcs/pull/77) +- [App Image Extensions (OS packages)](https://github.com/buildpacks/rfcs/pull/23) + +# Prior Art +[prior-art]: #prior-art + +- [RFC-0006: Stage specific mixins](https://github.com/buildpacks/rfcs/blob/main/text/0006-stage-specific-mixins.md) + +# Unresolved Questions +[unresolved-questions]: #unresolved-questions + +- Can users explicitly specify that they want a particular stackpack to run, or are mixins the only interface? +- What is a word that means "not-idempotent"? + +# Spec. Changes (OPTIONAL) +[spec-changes]: #spec-changes + + +## Stackpacks + +Stackpacks are identical to other buildpacks, with the following exceptions: + +1. The `` directory is NOT writable +1. The working directory WILL NOT contain application source code. +1. All changes made to the filesystem (with the exception of `/tmp`) during the execution of the stackpack's `bin/build` will be snaphotted and stored as a single layer. +1. A `launch.toml` WILL NOT be honored. + +## Build Plan (TOML) + +``` +[[mixins]] +type = "" +name = "" +``` + +Where: + +* `type` - the type of mixin: either `package` or `set` +* `name` - a pattern used to match mixin names + +## `buildpack.toml` + + This proposal adds a new key to the `[buildpack]` table in `buildpack.toml`: + + ``` + [buildpack] + privileged = + not-idempotent = + ``` + +* `privileged` - when set to `true`, the lifecycle will run this buildpack as the `root` user. +* `not-idempotent` - when set to `true`, indicates that the buildpack is not idempotent. The lifecycle will provide a clean filesystem from the stack image(s) before each run (i.e. no cache). \ No newline at end of file From c4aee7e88294d66f70c2575abeec880eb79d6f22 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Mon, 22 Jun 2020 11:40:34 -0500 Subject: [PATCH 002/140] Add RFC for App Mixins Signed-off-by: Joe Kutner --- text/0000-app-mixins.md | 78 +++++++++++++++++++++++++++-------------- 1 file changed, 52 insertions(+), 26 deletions(-) diff --git a/text/0000-app-mixins.md b/text/0000-app-mixins.md index 0115b6c9b..d3e52185b 100644 --- a/text/0000-app-mixins.md +++ b/text/0000-app-mixins.md @@ -65,14 +65,33 @@ The stackpack interface is identical to the buildpack interface (i.e. the same ` For each stackpack, the lifecycle will use [snapshotting](https://github.com/GoogleContainerTools/kaniko/blob/master/docs/designdoc.md#snapshotting-snapshotting) to capture changes made during the stackpack's build phase (excluding `/tmp`, `/cnb`, and `/layers`). Alternatively, a platform may store a new stack image to cache the changes. All of the captured changes will be included in a single layer produced as output from the stackpack. The `/layers` dir MAY NOT be used to create arbitrary layers. -A stackpack is included in a builder by defining it in the `builder.toml` like any other buildpack: +A stackpack is included in a builder by defining it in the `builder.toml` in the `[[stack.buildpacks]]` array of tables: ``` -[[buildpacks]] +[[stack.buildpacks]] id = "" uri = "" ``` +Each stackpack included in the builder will execute until all of a project's mixins have been provided. + +The stackpack's snapshot layer may be modified by writing a `launch.toml` file. The `[[processes]]` and `[[slices]]` tables may be used as normal, and a new `[[excludes]]` table will be introduced with the following schema: + +``` +[[excludes]] +paths = [""] +cache = false +``` + +Where: + +* `paths` = a list of paths to exclude from the layer +* `cache` = if true, the paths will be excluded from the launch image layer, but will be included in the cache layer. + +## Rebasing an App + +App that uses stackpacks can be rebased as normal. Stackpacks are expected to retain ABI compatibility. + ## Example: Apt Buildpack A buildpack that can install an arbitrary list of mixins would have a `buildpack.toml` like this: @@ -81,6 +100,9 @@ uri = "" [buildpack] id = "example/apt" privileged = true + +[[stacks]] +id = "io.buildpacks.stacks.bionic" ``` Its `bin/detect` would have the following contents: @@ -90,8 +112,7 @@ Its `bin/detect` would have the following contents: cat <"$2" [[mixins]] -type = "package" -name = "*" +name = "[^=]+" TOML ``` @@ -106,25 +127,13 @@ for package in $(cat $3 | yj -t | jq -r ".entries | .[] | .name"); do apt install $package done -rm -rf /var/cache/apt/archives +cat << EOF > launch.toml +[[excludes]] +paths = [ "/var/cache" ] +cache = true +EOF ``` -## Upgrading an App - -When attempting to rebase an image that used a stackpack to provide a mixin, the platform will attempt to re-run the stackpack(s) during rebase. - -When the base image has not changed -1. If the buildpacks are configure as idempotent (the default) load the previous layers onto the base images. -1. Run the buildpacks, creating an ephemeral image. -1. Rebase the app on to the new ephemeral image. - -When there is an update to either the build or run base images -1. Pull the new base image(s) -1. Run the root buildpacks against the new image(s) without loading previous layers, and create an ephemeral image. -1. Rebase the app on to the new ephemeral image. - -**Question**: should some buildpacks be able to opt-out of this rebase behavior (i.e. they are able to guarantee ABI compatibility)? - # Drawbacks [drawbacks]: #drawbacks @@ -144,8 +153,12 @@ When there is an update to either the build or run base images # Unresolved Questions [unresolved-questions]: #unresolved-questions -- Can users explicitly specify that they want a particular stackpack to run, or are mixins the only interface? +- How should a buildpack be identified as a stackpack? + - a special `key` in the `buildpack.toml`? + - a `stackpack.toml`? - What is a word that means "not-idempotent"? +- How do we exclude/include from the cache? +- Should the stackpack's detect have read-only access to the app? # Spec. Changes (OPTIONAL) [spec-changes]: #spec-changes @@ -160,20 +173,33 @@ Stackpacks are identical to other buildpacks, with the following exceptions: 1. All changes made to the filesystem (with the exception of `/tmp`) during the execution of the stackpack's `bin/build` will be snaphotted and stored as a single layer. 1. A `launch.toml` WILL NOT be honored. +## launch.toml (TOML) + +``` +[[excludes]] +paths = [""] +cache = false +``` + +Where: + +* `paths` = a list of paths to exclude from the layer +* `cache` = if true, the paths will be excluded from the launch image layer, but will be included in the cache layer. + ## Build Plan (TOML) ``` [[mixins]] -type = "" name = "" +type = "" ``` Where: -* `type` - the type of mixin: either `package` or `set` -* `name` - a pattern used to match mixin names +* `type` - (default=`provides`) whether or not the mixin will be provided or required. +* `name` - a pattern used to match mixin names. adheres to [re2 syntax](https://github.com/google/re2/wiki/Syntax). -## `buildpack.toml` +## buildpack.toml (TOML) This proposal adds a new key to the `[buildpack]` table in `buildpack.toml`: From ee94de755206e43bb2fdebf75d12cdaebc6001b7 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Wed, 1 Jul 2020 08:56:44 -0500 Subject: [PATCH 003/140] double down on non-idempotent as a key in buildpack.toml Signed-off-by: Joe Kutner --- text/0000-app-mixins.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/text/0000-app-mixins.md b/text/0000-app-mixins.md index d3e52185b..b0cea2271 100644 --- a/text/0000-app-mixins.md +++ b/text/0000-app-mixins.md @@ -149,6 +149,7 @@ EOF [prior-art]: #prior-art - [RFC-0006: Stage specific mixins](https://github.com/buildpacks/rfcs/blob/main/text/0006-stage-specific-mixins.md) +- The term "non-idempotent" is used in section 9.1.2 of [Hypertext Transfer Protocol -- HTTP/1.1 (RFC 2616)](https://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html) # Unresolved Questions [unresolved-questions]: #unresolved-questions @@ -156,7 +157,6 @@ EOF - How should a buildpack be identified as a stackpack? - a special `key` in the `buildpack.toml`? - a `stackpack.toml`? -- What is a word that means "not-idempotent"? - How do we exclude/include from the cache? - Should the stackpack's detect have read-only access to the app? @@ -206,8 +206,8 @@ Where: ``` [buildpack] privileged = - not-idempotent = + non-idempotent = ``` * `privileged` - when set to `true`, the lifecycle will run this buildpack as the `root` user. -* `not-idempotent` - when set to `true`, indicates that the buildpack is not idempotent. The lifecycle will provide a clean filesystem from the stack image(s) before each run (i.e. no cache). \ No newline at end of file +* `non-idempotent` - when set to `true`, indicates that the buildpack is not idempotent. The lifecycle will provide a clean filesystem from the stack image(s) before each run (i.e. no cache). \ No newline at end of file From 7263b8f81305d9f6ffd51be01b7a8405bfbad804 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Wed, 1 Jul 2020 08:59:10 -0500 Subject: [PATCH 004/140] revised open questions Signed-off-by: Joe Kutner --- text/0000-app-mixins.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/text/0000-app-mixins.md b/text/0000-app-mixins.md index b0cea2271..38235f4b9 100644 --- a/text/0000-app-mixins.md +++ b/text/0000-app-mixins.md @@ -155,10 +155,11 @@ EOF [unresolved-questions]: #unresolved-questions - How should a buildpack be identified as a stackpack? + - Not everyone likes `privileged` - a special `key` in the `buildpack.toml`? - a `stackpack.toml`? -- How do we exclude/include from the cache? - Should the stackpack's detect have read-only access to the app? + - In this proposal it does, and in the future a more generic "root buildpack" construct would need it too. # Spec. Changes (OPTIONAL) [spec-changes]: #spec-changes From 79b061c8e2a7aca3b04eba00667b12c5d6837fba Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Mon, 6 Jul 2020 08:38:49 -0500 Subject: [PATCH 005/140] Define static mixins in stackpack Signed-off-by: Joe Kutner --- text/0000-app-mixins.md | 29 ++++++++--------------------- 1 file changed, 8 insertions(+), 21 deletions(-) diff --git a/text/0000-app-mixins.md b/text/0000-app-mixins.md index 38235f4b9..b03c68b17 100644 --- a/text/0000-app-mixins.md +++ b/text/0000-app-mixins.md @@ -110,10 +110,7 @@ Its `bin/detect` would have the following contents: ```bash #!/usr/bin/env bash -cat <"$2" -[[mixins]] -name = "[^=]+" -TOML +exit 0 ``` Its `bin/build` would have the following contents: @@ -187,27 +184,17 @@ Where: * `paths` = a list of paths to exclude from the layer * `cache` = if true, the paths will be excluded from the launch image layer, but will be included in the cache layer. -## Build Plan (TOML) - -``` -[[mixins]] -name = "" -type = "" -``` - -Where: - -* `type` - (default=`provides`) whether or not the mixin will be provided or required. -* `name` - a pattern used to match mixin names. adheres to [re2 syntax](https://github.com/google/re2/wiki/Syntax). - ## buildpack.toml (TOML) - This proposal adds a new key to the `[buildpack]` table in `buildpack.toml`: + This proposal adds new keys to the `[buildpack]` table in `buildpack.toml`, and a new `[[mixins]]` array of tables: ``` - [buildpack] - privileged = - non-idempotent = +[buildpack] +privileged = +non-idempotent = + +[[mixins]] +name = "" ``` * `privileged` - when set to `true`, the lifecycle will run this buildpack as the `root` user. From 9f3957208afce95802e5c3140a8bf23bd4661a7b Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Mon, 6 Jul 2020 11:17:22 -0500 Subject: [PATCH 006/140] Revisions from PR feedback Signed-off-by: Joe Kutner --- text/0000-app-mixins.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/text/0000-app-mixins.md b/text/0000-app-mixins.md index b03c68b17..2f692984c 100644 --- a/text/0000-app-mixins.md +++ b/text/0000-app-mixins.md @@ -24,7 +24,7 @@ Mixins already allow buildpack authors to create buildpacks that depend on an ex - *application developer* - a person who is using buildpacks to transform their application code into an OCI image - *stackpack* - a new type of buildpack that runs against the stack image(s) instead of an app -A application developer may specify a list of mixins in their application's `project.toml` file like this: +An application developer may specify a list of mixins in their application's `project.toml` file like this: ```toml [build] @@ -169,7 +169,6 @@ Stackpacks are identical to other buildpacks, with the following exceptions: 1. The `` directory is NOT writable 1. The working directory WILL NOT contain application source code. 1. All changes made to the filesystem (with the exception of `/tmp`) during the execution of the stackpack's `bin/build` will be snaphotted and stored as a single layer. -1. A `launch.toml` WILL NOT be honored. ## launch.toml (TOML) From 8c92fd13e79e771188605b88f47c287c9141cbf0 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Mon, 6 Jul 2020 13:33:50 -0500 Subject: [PATCH 007/140] Revise rebase description in stackpacks Signed-off-by: Joe Kutner --- text/0000-app-mixins.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-app-mixins.md b/text/0000-app-mixins.md index 2f692984c..34bcb555b 100644 --- a/text/0000-app-mixins.md +++ b/text/0000-app-mixins.md @@ -90,7 +90,7 @@ Where: ## Rebasing an App -App that uses stackpacks can be rebased as normal. Stackpacks are expected to retain ABI compatibility. +Before a launch image is rebased, the platform must re-run the any stackpacks that were used to build the launch image against the new run-image. Then, the rebase operation can be performed as normal, while including the stackpack layers as part of the stack. This will be made possible by including the stackpack in the run-image, but because the stackpack detect phase is not run, the operation does not need access to the application source. ## Example: Apt Buildpack From 60ab9e206dc30db698ecb4a767dade2f125c6d38 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Mon, 6 Jul 2020 13:40:32 -0500 Subject: [PATCH 008/140] Revisions to spec changes for stackpacks Signed-off-by: Joe Kutner --- text/0000-app-mixins.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/text/0000-app-mixins.md b/text/0000-app-mixins.md index 34bcb555b..f1312d591 100644 --- a/text/0000-app-mixins.md +++ b/text/0000-app-mixins.md @@ -161,13 +161,12 @@ EOF # Spec. Changes (OPTIONAL) [spec-changes]: #spec-changes - ## Stackpacks Stackpacks are identical to other buildpacks, with the following exceptions: 1. The `` directory is NOT writable -1. The working directory WILL NOT contain application source code. +1. The working directory WILL NOT contain application source code during the build phase. 1. All changes made to the filesystem (with the exception of `/tmp`) during the execution of the stackpack's `bin/build` will be snaphotted and stored as a single layer. ## launch.toml (TOML) @@ -197,4 +196,8 @@ name = "" ``` * `privileged` - when set to `true`, the lifecycle will run this buildpack as the `root` user. -* `non-idempotent` - when set to `true`, indicates that the buildpack is not idempotent. The lifecycle will provide a clean filesystem from the stack image(s) before each run (i.e. no cache). \ No newline at end of file +* `non-idempotent` - when set to `true`, indicates that the buildpack is not idempotent. The lifecycle will provide a clean filesystem from the stack image(s) before each run (i.e. no cache). + +The `[[mixins]]` array of tables defines the list of mixins provided by this buildpack. + +* `name` - a pattern describing the mixins provide by this buildpack \ No newline at end of file From d5af5864173983285ecd8339cd7e5693c89d1920 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Mon, 6 Jul 2020 13:42:43 -0500 Subject: [PATCH 009/140] Revise open questions for stackpacks Signed-off-by: Joe Kutner --- text/0000-app-mixins.md | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/text/0000-app-mixins.md b/text/0000-app-mixins.md index f1312d591..0473c0bed 100644 --- a/text/0000-app-mixins.md +++ b/text/0000-app-mixins.md @@ -151,12 +151,8 @@ EOF # Unresolved Questions [unresolved-questions]: #unresolved-questions -- How should a buildpack be identified as a stackpack? - - Not everyone likes `privileged` - - a special `key` in the `buildpack.toml`? - - a `stackpack.toml`? - Should the stackpack's detect have read-only access to the app? - - In this proposal it does, and in the future a more generic "root buildpack" construct would need it too. + - This would likely be driven by a stackpack that does not provide mixins, but instead dynamically contributes to the build plan based on the contents of the app source code. For example, a stackpack that adds custom cacerts. # Spec. Changes (OPTIONAL) [spec-changes]: #spec-changes From 6f95c0789804c7cca041c17b3f866c5ff669c7ae Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Mon, 6 Jul 2020 13:45:56 -0500 Subject: [PATCH 010/140] Define root buildpack Signed-off-by: Joe Kutner --- text/0000-app-mixins.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/text/0000-app-mixins.md b/text/0000-app-mixins.md index 0473c0bed..e4a8fe7ac 100644 --- a/text/0000-app-mixins.md +++ b/text/0000-app-mixins.md @@ -22,7 +22,8 @@ Mixins already allow buildpack authors to create buildpacks that depend on an ex [what-it-is]: #what-it-is - *application developer* - a person who is using buildpacks to transform their application code into an OCI image -- *stackpack* - a new type of buildpack that runs against the stack image(s) instead of an app +- *root buildpack* - a new type of buildpack that runs as the root user +- *stackpack* - a type of root buildpack that runs against the stack image(s) instead of an app An application developer may specify a list of mixins in their application's `project.toml` file like this: @@ -141,6 +142,7 @@ EOF - [Add root buildpack as interface for app image extension RFC](https://github.com/buildpacks/rfcs/pull/77) - [App Image Extensions (OS packages)](https://github.com/buildpacks/rfcs/pull/23) +- Allow application developers to use root buildpacks in `project.toml`. This would have significant performance implications, and creates a "foot gun" in qhich end users could build images they are not able to maintain. For this reason, we are holding off on a generic root buildpack feature. # Prior Art [prior-art]: #prior-art From da63c101e8d271c4d8d8f05b31afe1404d2e9c90 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Mon, 6 Jul 2020 13:47:26 -0500 Subject: [PATCH 011/140] Link root buildpack Signed-off-by: Joe Kutner --- text/0000-app-mixins.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-app-mixins.md b/text/0000-app-mixins.md index e4a8fe7ac..bbb99b8db 100644 --- a/text/0000-app-mixins.md +++ b/text/0000-app-mixins.md @@ -142,7 +142,7 @@ EOF - [Add root buildpack as interface for app image extension RFC](https://github.com/buildpacks/rfcs/pull/77) - [App Image Extensions (OS packages)](https://github.com/buildpacks/rfcs/pull/23) -- Allow application developers to use root buildpacks in `project.toml`. This would have significant performance implications, and creates a "foot gun" in qhich end users could build images they are not able to maintain. For this reason, we are holding off on a generic root buildpack feature. +- [Root Buildpacks](https://github.com/buildpacks/rfcs/blob/root-buildpacks/text/0000-root-buildpacks.md): Allow application developers to use root buildpacks in `project.toml`. This would have significant performance implications, and creates a "foot gun" in qhich end users could build images they are not able to maintain. For this reason, we are holding off on a generic root buildpack feature. # Prior Art [prior-art]: #prior-art From 3f4b887ef8042363a89a6b76b1f2de44a73880ea Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Mon, 6 Jul 2020 13:50:10 -0500 Subject: [PATCH 012/140] Move buildpack mixins to list key Signed-off-by: Joe Kutner --- text/0000-app-mixins.md | 9 ++------- 1 file changed, 2 insertions(+), 7 deletions(-) diff --git a/text/0000-app-mixins.md b/text/0000-app-mixins.md index bbb99b8db..8f669665f 100644 --- a/text/0000-app-mixins.md +++ b/text/0000-app-mixins.md @@ -188,14 +188,9 @@ Where: [buildpack] privileged = non-idempotent = - -[[mixins]] -name = "" +mixins = [ "" ] ``` * `privileged` - when set to `true`, the lifecycle will run this buildpack as the `root` user. * `non-idempotent` - when set to `true`, indicates that the buildpack is not idempotent. The lifecycle will provide a clean filesystem from the stack image(s) before each run (i.e. no cache). - -The `[[mixins]]` array of tables defines the list of mixins provided by this buildpack. - -* `name` - a pattern describing the mixins provide by this buildpack \ No newline at end of file +* `mixins` - a list of patterns that match mixins provided by this buildpack \ No newline at end of file From af26a3b3a5dec4939cad00b5b172674de3dca1a5 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Mon, 6 Jul 2020 13:51:41 -0500 Subject: [PATCH 013/140] Update stackpack example with mixins list Signed-off-by: Joe Kutner --- text/0000-app-mixins.md | 1 + 1 file changed, 1 insertion(+) diff --git a/text/0000-app-mixins.md b/text/0000-app-mixins.md index 8f669665f..56e4e64e5 100644 --- a/text/0000-app-mixins.md +++ b/text/0000-app-mixins.md @@ -101,6 +101,7 @@ Before a launch image is rebased, the platform must re-run the any stackpacks th [buildpack] id = "example/apt" privileged = true +mixins = ["[^=]+"] [[stacks]] id = "io.buildpacks.stacks.bionic" From 18eea7cf48d1ac86368868d66ac65ae3c73beba4 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Mon, 6 Jul 2020 14:09:33 -0500 Subject: [PATCH 014/140] Use mixins to provide custom cacerts Signed-off-by: Joe Kutner --- text/0000-app-mixins.md | 48 +++++++++++++++++++++++++++++++++++++++-- 1 file changed, 46 insertions(+), 2 deletions(-) diff --git a/text/0000-app-mixins.md b/text/0000-app-mixins.md index 56e4e64e5..399bc4dae 100644 --- a/text/0000-app-mixins.md +++ b/text/0000-app-mixins.md @@ -95,7 +95,7 @@ Before a launch image is rebased, the platform must re-run the any stackpacks th ## Example: Apt Buildpack - A buildpack that can install an arbitrary list of mixins would have a `buildpack.toml` like this: +A buildpack that installs a custom certificate would have a `buildpack.toml` like this: ```toml [buildpack] @@ -133,6 +133,49 @@ cache = true EOF ``` +## Example: CA Certificate Buildpack + +A buildpack that works on the `ubuntu-20` stack, which defines a `type=` prefix for mixins, and installs customer CA Certificates would have a `buildpack.toml` that looks like this: + +```toml +[buildpack] +id = "example/cacerts" +privileged = true +mixins = ["type=cacert"] + +[[stacks]] +id = "ubuntu-20" +``` + +Its `bin/detect` would have the following contents: + + ```bash +#!/usr/bin/env bash + +exit 0 +``` + +Its `bin/build` would have the following contents: + + ```bash +#!/usr/bin/env bash + +for cert_path in $(cat $3 | yj -t | jq -r ".entries | .[] | .name"); do + cert_file=$(basename $cert_path) + cp $cert_path /usr/share/ca-certificates/$cert_file + chmod 644 /usr/share/ca-certificates/$cert_file +done + +update-ca-certificates +``` + +An application that is using this stackpack would the following `project.toml`: + +```toml +[build] +mixins = [ "type=cacert:mycerts/database.crt" ] +``` + # Drawbacks [drawbacks]: #drawbacks @@ -155,7 +198,8 @@ EOF [unresolved-questions]: #unresolved-questions - Should the stackpack's detect have read-only access to the app? - - This would likely be driven by a stackpack that does not provide mixins, but instead dynamically contributes to the build plan based on the contents of the app source code. For example, a stackpack that adds custom cacerts. + - Does a stackpack even need a detect phase? + - This would likely be driven by a stackpack that does not provide mixins, but instead dynamically contributes to the build plan based on the contents of the app source code. I don't know if we have a use case for this, but I can imaging a buildpack that reads environment variables as input to some function. # Spec. Changes (OPTIONAL) [spec-changes]: #spec-changes From 4ce2962471760dbcaf43d9e5c320cb2479386a1a Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Tue, 7 Jul 2020 12:37:32 -0500 Subject: [PATCH 015/140] Update text/0000-app-mixins.md Signed-off-by: Joe Kutner Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- text/0000-app-mixins.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/text/0000-app-mixins.md b/text/0000-app-mixins.md index 399bc4dae..58079c130 100644 --- a/text/0000-app-mixins.md +++ b/text/0000-app-mixins.md @@ -210,7 +210,7 @@ Stackpacks are identical to other buildpacks, with the following exceptions: 1. The `` directory is NOT writable 1. The working directory WILL NOT contain application source code during the build phase. -1. All changes made to the filesystem (with the exception of `/tmp`) during the execution of the stackpack's `bin/build` will be snaphotted and stored as a single layer. +1. All changes made to the filesystem (with the exception of `/tmp`) during the execution of the stackpack's `bin/build` will be snapshotted and stored as a single layer. ## launch.toml (TOML) @@ -238,4 +238,4 @@ mixins = [ "" ] * `privileged` - when set to `true`, the lifecycle will run this buildpack as the `root` user. * `non-idempotent` - when set to `true`, indicates that the buildpack is not idempotent. The lifecycle will provide a clean filesystem from the stack image(s) before each run (i.e. no cache). -* `mixins` - a list of patterns that match mixins provided by this buildpack \ No newline at end of file +* `mixins` - a list of patterns that match mixins provided by this buildpack From 9f2210a26ccd0aa8920a32c808dd8526a4a2156a Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Tue, 7 Jul 2020 12:37:39 -0500 Subject: [PATCH 016/140] Update text/0000-app-mixins.md Signed-off-by: Joe Kutner Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- text/0000-app-mixins.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-app-mixins.md b/text/0000-app-mixins.md index 58079c130..e440e74b7 100644 --- a/text/0000-app-mixins.md +++ b/text/0000-app-mixins.md @@ -208,7 +208,7 @@ mixins = [ "type=cacert:mycerts/database.crt" ] Stackpacks are identical to other buildpacks, with the following exceptions: -1. The `` directory is NOT writable +1. The `` directory is NOT writable. 1. The working directory WILL NOT contain application source code during the build phase. 1. All changes made to the filesystem (with the exception of `/tmp`) during the execution of the stackpack's `bin/build` will be snapshotted and stored as a single layer. From a1f10380ede446b6c3b3c80b5f841e5284fbc9b0 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Tue, 7 Jul 2020 12:37:51 -0500 Subject: [PATCH 017/140] Update text/0000-app-mixins.md Signed-off-by: Joe Kutner Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- text/0000-app-mixins.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-app-mixins.md b/text/0000-app-mixins.md index e440e74b7..86bdd5ce5 100644 --- a/text/0000-app-mixins.md +++ b/text/0000-app-mixins.md @@ -186,7 +186,7 @@ mixins = [ "type=cacert:mycerts/database.crt" ] - [Add root buildpack as interface for app image extension RFC](https://github.com/buildpacks/rfcs/pull/77) - [App Image Extensions (OS packages)](https://github.com/buildpacks/rfcs/pull/23) -- [Root Buildpacks](https://github.com/buildpacks/rfcs/blob/root-buildpacks/text/0000-root-buildpacks.md): Allow application developers to use root buildpacks in `project.toml`. This would have significant performance implications, and creates a "foot gun" in qhich end users could build images they are not able to maintain. For this reason, we are holding off on a generic root buildpack feature. +- [Root Buildpacks](https://github.com/buildpacks/rfcs/blob/root-buildpacks/text/0000-root-buildpacks.md): Allow application developers to use root buildpacks in `project.toml`. This would have significant performance implications, and creates a "foot gun" in which end users could build images they are not able to maintain. For this reason, we are holding off on a generic root buildpack feature. # Prior Art [prior-art]: #prior-art From 70ba98507fc413747998d5ae23eb4c2980ba980a Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Tue, 7 Jul 2020 12:37:59 -0500 Subject: [PATCH 018/140] Update text/0000-app-mixins.md Signed-off-by: Joe Kutner Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- text/0000-app-mixins.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-app-mixins.md b/text/0000-app-mixins.md index 86bdd5ce5..fe64847b7 100644 --- a/text/0000-app-mixins.md +++ b/text/0000-app-mixins.md @@ -179,7 +179,7 @@ mixins = [ "type=cacert:mycerts/database.crt" ] # Drawbacks [drawbacks]: #drawbacks -- Mixins are less flexible that a generic [root buildpack](https://github.com/buildpacks/rfcs/blob/app-image-ext-buildpacks/text/0000-app-image-extensions.md). +- Mixins are less flexible than a generic [root buildpack](https://github.com/buildpacks/rfcs/blob/app-image-ext-buildpacks/text/0000-app-image-extensions.md). # Alternatives [alternatives]: #alternatives From efb1fa600ad7434f378273699b072d1d2679616a Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Tue, 7 Jul 2020 12:42:34 -0500 Subject: [PATCH 019/140] Fix description of Apt stackpack example Signed-off-by: Joe Kutner --- text/0000-app-mixins.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-app-mixins.md b/text/0000-app-mixins.md index fe64847b7..068df567f 100644 --- a/text/0000-app-mixins.md +++ b/text/0000-app-mixins.md @@ -95,7 +95,7 @@ Before a launch image is rebased, the platform must re-run the any stackpacks th ## Example: Apt Buildpack -A buildpack that installs a custom certificate would have a `buildpack.toml` like this: +A buildpack that can install an arbitrary list of mixins would have a `buildpack.toml` like this: ```toml [buildpack] From 161e6b8bd642fced0c205b6d5b3fcb3ae8f5c320 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Tue, 7 Jul 2020 12:43:57 -0500 Subject: [PATCH 020/140] Update text/0000-app-mixins.md Signed-off-by: Joe Kutner Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- text/0000-app-mixins.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/text/0000-app-mixins.md b/text/0000-app-mixins.md index 068df567f..68ccdd1d8 100644 --- a/text/0000-app-mixins.md +++ b/text/0000-app-mixins.md @@ -44,7 +44,7 @@ When a command like `pack build` is run, the list of mixins will be processed be When a list of mixins are defined in a `project.toml` and `pack build` is run against that project, the following will happen: 1. The lifecycle will compare the list of mixins to those provided by the stack. If all mixin names are provided by the stack, no further action is required. -1. In any requested mixin is not provided by the stack, the lifecycle will run the detect phase for all stackpacks defined in the builder. +1. If any requested mixin is not provided by the stack, the lifecycle will run the detect phase for all stackpacks defined in the builder. 1. The lifecycle will compare the mixins added to the build plan to see if they match they required mixins. 1. If at least one stackpack passes detect and provides the required mixin(s), the lifecycle will execute the stackpack build phase for passing stackpack(s). If no stackpacks pass detect, or no stackpacks provide the required mixin, the build will fail with a message that describes the mixins that could not be provided. 1. During the lifecycle's build phase, the stackpacks that passed detection will run against the build and run images accordingly (see details below). All stackpacks will be run before the user's buildpacks. @@ -135,7 +135,7 @@ EOF ## Example: CA Certificate Buildpack -A buildpack that works on the `ubuntu-20` stack, which defines a `type=` prefix for mixins, and installs customer CA Certificates would have a `buildpack.toml` that looks like this: +A buildpack that works on the `ubuntu-20` stack, which defines a `type=` prefix for mixins, and installs custom CA Certificates would have a `buildpack.toml` that looks like this: ```toml [buildpack] From 4d21e95d6a642eb04325c8c2023de92e04760a98 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Mon, 13 Jul 2020 10:49:40 -0500 Subject: [PATCH 021/140] Update text/0000-app-mixins.md Signed-off-by: Joe Kutner Co-authored-by: Stephen Levine --- text/0000-app-mixins.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-app-mixins.md b/text/0000-app-mixins.md index 68ccdd1d8..f4ac3108b 100644 --- a/text/0000-app-mixins.md +++ b/text/0000-app-mixins.md @@ -41,7 +41,7 @@ When a command like `pack build` is run, the list of mixins will be processed be # How it Works [how-it-works]: #how-it-works -When a list of mixins are defined in a `project.toml` and `pack build` is run against that project, the following will happen: +When a list of mixins are required by buildpacks via the build plan and the build phase starts: 1. The lifecycle will compare the list of mixins to those provided by the stack. If all mixin names are provided by the stack, no further action is required. 1. If any requested mixin is not provided by the stack, the lifecycle will run the detect phase for all stackpacks defined in the builder. From 1f53e72b2a5fbc3b11dedae2c7e5f2fa28efbf61 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Mon, 13 Jul 2020 10:49:54 -0500 Subject: [PATCH 022/140] Update text/0000-app-mixins.md Signed-off-by: Joe Kutner Co-authored-by: Stephen Levine --- text/0000-app-mixins.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-app-mixins.md b/text/0000-app-mixins.md index f4ac3108b..ba8933c20 100644 --- a/text/0000-app-mixins.md +++ b/text/0000-app-mixins.md @@ -59,7 +59,7 @@ When a list of mixins are required by buildpacks via the build plan and the buil * Does not have access to the application source code * Is run before all regular buildpacks * Is run against both the build and run images -* Is distributed with the builder image +* Is distributed with the stack run and/or build image * May not write to the `/layers` directory The stackpack interface is identical to the buildpack interface (i.e. the same `bin/detect` and `bin/build` scripts are required). However, some of the context it is run in is different from regular buildpacks. From f4bf73d26912d9101cdcb8bd1c109d6fb585999f Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Mon, 13 Jul 2020 10:50:20 -0500 Subject: [PATCH 023/140] Update text/0000-app-mixins.md Signed-off-by: Joe Kutner Co-authored-by: Stephen Levine --- text/0000-app-mixins.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/text/0000-app-mixins.md b/text/0000-app-mixins.md index ba8933c20..71d178d7e 100644 --- a/text/0000-app-mixins.md +++ b/text/0000-app-mixins.md @@ -16,7 +16,9 @@ This is a proposal for allowing application developers (buildpack users) to spec # Motivation [motivation]: #motivation -Mixins already allow buildpack authors to create buildpacks that depend on an extended set of OS packages without affecting build time, but it's common for application code to depend on OS packages too. But allowing buildpacks to arbitrarily install OS packages during build would drastically increase build time, especially when CNB tooling is used to build or rebase many apps with similar package requirements. For that reason, this proposal defines a mechanism to implement mixin support for application developers without significant impact on build performance. +Mixins already allow buildpack authors to create buildpacks that depend on an extended set of OS packages without affecting build time, but it's common for application code to depend on OS packages too. But allowing buildpacks to arbitrarily install OS packages during build would drastically increase build time, especially when CNB tooling is used to build or rebase many apps with similar package requirements. + +For that reason, this proposal defines a mechanism to implement dynamic installation of mixins at build time while minimizing the performance impact. This is accomplished by allowing stack images to satisfy mixin requires statically and/or reject dynamic mixin installation early in the build process. # What it is [what-it-is]: #what-it-is From cb197599e32c35a18f41e5d09f131c836dcb86fa Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Mon, 13 Jul 2020 10:52:11 -0500 Subject: [PATCH 024/140] Update text/0000-app-mixins.md Signed-off-by: Joe Kutner Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- text/0000-app-mixins.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-app-mixins.md b/text/0000-app-mixins.md index 71d178d7e..f8b21f34a 100644 --- a/text/0000-app-mixins.md +++ b/text/0000-app-mixins.md @@ -48,7 +48,7 @@ When a list of mixins are required by buildpacks via the build plan and the buil 1. The lifecycle will compare the list of mixins to those provided by the stack. If all mixin names are provided by the stack, no further action is required. 1. If any requested mixin is not provided by the stack, the lifecycle will run the detect phase for all stackpacks defined in the builder. 1. The lifecycle will compare the mixins added to the build plan to see if they match they required mixins. -1. If at least one stackpack passes detect and provides the required mixin(s), the lifecycle will execute the stackpack build phase for passing stackpack(s). If no stackpacks pass detect, or no stackpacks provide the required mixin, the build will fail with a message that describes the mixins that could not be provided. +1. If at least one stackpack passes detect and provides the required mixin(s), the lifecycle will execute the stackpack build phase for all passing stackpack(s). If no stackpacks pass detect, or no stackpacks provide the required mixin, the build will fail with a message that describes the mixins that could not be provided. 1. During the lifecycle's build phase, the stackpacks that passed detection will run against the build and run images accordingly (see details below). All stackpacks will be run before the user's buildpacks. ## Stackpacks From c5739debdc495a760f33d46af189ad2744da241f Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Mon, 13 Jul 2020 10:54:29 -0500 Subject: [PATCH 025/140] Reword how stackpacks are used by the builder Signed-off-by: Joe Kutner --- text/0000-app-mixins.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-app-mixins.md b/text/0000-app-mixins.md index f8b21f34a..afa8e1c32 100644 --- a/text/0000-app-mixins.md +++ b/text/0000-app-mixins.md @@ -68,7 +68,7 @@ The stackpack interface is identical to the buildpack interface (i.e. the same ` For each stackpack, the lifecycle will use [snapshotting](https://github.com/GoogleContainerTools/kaniko/blob/master/docs/designdoc.md#snapshotting-snapshotting) to capture changes made during the stackpack's build phase (excluding `/tmp`, `/cnb`, and `/layers`). Alternatively, a platform may store a new stack image to cache the changes. All of the captured changes will be included in a single layer produced as output from the stackpack. The `/layers` dir MAY NOT be used to create arbitrary layers. -A stackpack is included in a builder by defining it in the `builder.toml` in the `[[stack.buildpacks]]` array of tables: +A builder can use a stackpack by defining it in the `builder.toml` in the `[[stack.buildpacks]]` array of tables: ``` [[stack.buildpacks]] From 00cfc80657900610a65fecec245b4ebffe2709ea Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Mon, 13 Jul 2020 10:56:01 -0500 Subject: [PATCH 026/140] Reword how stackpack execution is determined Signed-off-by: Joe Kutner --- text/0000-app-mixins.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-app-mixins.md b/text/0000-app-mixins.md index afa8e1c32..20a2b7b8d 100644 --- a/text/0000-app-mixins.md +++ b/text/0000-app-mixins.md @@ -76,7 +76,7 @@ id = "" uri = "" ``` -Each stackpack included in the builder will execute until all of a project's mixins have been provided. +Each stackpack used by the builder will execute if it passes detection and it's provided mixins satisfy one of a project's required mixins. The stackpack's snapshot layer may be modified by writing a `launch.toml` file. The `[[processes]]` and `[[slices]]` tables may be used as normal, and a new `[[excludes]]` table will be introduced with the following schema: From 1ecabdd4810f0156f0555accea111a151303d9e0 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Mon, 13 Jul 2020 10:57:45 -0500 Subject: [PATCH 027/140] Expand definition of stackpack Signed-off-by: Joe Kutner --- text/0000-app-mixins.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-app-mixins.md b/text/0000-app-mixins.md index 20a2b7b8d..1b84734ac 100644 --- a/text/0000-app-mixins.md +++ b/text/0000-app-mixins.md @@ -25,7 +25,7 @@ For that reason, this proposal defines a mechanism to implement dynamic installa - *application developer* - a person who is using buildpacks to transform their application code into an OCI image - *root buildpack* - a new type of buildpack that runs as the root user -- *stackpack* - a type of root buildpack that runs against the stack image(s) instead of an app +- *stackpack* - a type of root buildpack that runs against the stack image(s) instead of an app. It is distinguished by a static list of mixins it can provide. An application developer may specify a list of mixins in their application's `project.toml` file like this: From 325460e5aee3e9ae9472300952d4351da8acdc5f Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Thu, 16 Jul 2020 10:54:18 -0500 Subject: [PATCH 028/140] Added open questions to App Mixins Signed-off-by: Joe Kutner --- text/0000-app-mixins.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/text/0000-app-mixins.md b/text/0000-app-mixins.md index 1b84734ac..c2e4abe8d 100644 --- a/text/0000-app-mixins.md +++ b/text/0000-app-mixins.md @@ -202,6 +202,17 @@ mixins = [ "type=cacert:mycerts/database.crt" ] - Should the stackpack's detect have read-only access to the app? - Does a stackpack even need a detect phase? - This would likely be driven by a stackpack that does not provide mixins, but instead dynamically contributes to the build plan based on the contents of the app source code. I don't know if we have a use case for this, but I can imaging a buildpack that reads environment variables as input to some function. +- Should stackpacks be able to define per-stack mixins? + - We could support a top-level `mixins` array, and allow refinements under `[[stacks]] mixins`. If we do this, we need to distinguish between provided and required mixins (at least under `[[stacks]]`). + - If buildpacks can require mixins from `bin/detect`, the stackpack could use this mechanism to require per-stack mixins. +- Should we use regex to match mixins, or should we double-down on `type=` (or similar)? + - regexs make it likely that a stackpack will be unable to distinguish packages from other types of mixins + - It will be difficult for the platform to distinguish between regex patterns and explict value to match. + - Alternatively, we could define a URN for mixins, like: + - `urn:package:libpq` + - `urn:package:build:libpq-dev` + - `urn:cacert:mycerts/database.crt` + - Naked strings, like `libpq`, could be assumed to represent packages. # Spec. Changes (OPTIONAL) [spec-changes]: #spec-changes From 6a3cfba18c3b1f3687e1344d17e99e0ff27e95d4 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Fri, 17 Jul 2020 22:42:42 -0500 Subject: [PATCH 029/140] Update the cacert example Signed-off-by: Joe Kutner --- text/0000-app-mixins.md | 79 ++++++++++++++++++++++++++++++++--------- 1 file changed, 62 insertions(+), 17 deletions(-) diff --git a/text/0000-app-mixins.md b/text/0000-app-mixins.md index c2e4abe8d..33540684e 100644 --- a/text/0000-app-mixins.md +++ b/text/0000-app-mixins.md @@ -137,13 +137,17 @@ EOF ## Example: CA Certificate Buildpack +Support for custom CA Certificates can be accomplished with two buildpacks: a stackpack that can install the cert, and a normal buildpack that can provide a cert in the build plan. + +### CA Cert Installer Stackpack + A buildpack that works on the `ubuntu-20` stack, which defines a `type=` prefix for mixins, and installs custom CA Certificates would have a `buildpack.toml` that looks like this: ```toml [buildpack] id = "example/cacerts" privileged = true -mixins = ["type=cacert"] +mixins = ["cacert"] [[stacks]] id = "ubuntu-20" @@ -154,28 +158,76 @@ Its `bin/detect` would have the following contents: ```bash #!/usr/bin/env bash +cat << EOF > $2 +[[provides]] +name = "cacert" +EOF + exit 0 ``` -Its `bin/build` would have the following contents: +Its `bin/build` would would install each `cacert` in the build plan. It would have the following contents: ```bash #!/usr/bin/env bash -for cert_path in $(cat $3 | yj -t | jq -r ".entries | .[] | .name"); do - cert_file=$(basename $cert_path) - cp $cert_path /usr/share/ca-certificates/$cert_file - chmod 644 /usr/share/ca-certificates/$cert_file +# filter to the cert +for file in $(cat $3 | yj -t | jq -r ".entries | .[] | select(.name==\"cacert\") | .metadata | .file"); do + cert="$(cat $3 | yj -t | jq -r ".entries | .[] | select(.name==\"cacert\") | .metadata | select(.file==\"$file\") | .content")" + echo $cert > /usr/share/ca-certificates/$file + chmod 644 /usr/share/ca-certificates/$file done update-ca-certificates ``` -An application that is using this stackpack would the following `project.toml`: +### CA Cert Provider Buildpack + +The stackpack must be used with a buildpack that provides a certificate(s) for it to install. That buildpack would have the following `buildpack.toml`: ```toml -[build] -mixins = [ "type=cacert:mycerts/database.crt" ] +[buildpack] +id = "my/db-cert" + +[[stacks]] +id = "ubuntu-20" +mixins = ["cacert"] +``` + +Its `bin/detect` would require a certificate with the following contents: + + ```bash +#!/usr/bin/env bash + +cat << EOF > $2 +[[requires]] +name = "cacert" + +[requires.metadata] +file = "database.crt" +content = """ +$(cat $CNB_BUILDPACK_DIR/database.crt) +""" + +[[requires]] +name = "cacert" + +[requires.metadata] +file = "server.crt" +content = """ +$(cat $CNB_BUILDPACK_DIR/server.crt) +""" +EOF + +exit 0 +``` + +Its `bin/build` would do nothing, and `have the following contents: + + ```bash +#!/usr/bin/env bash + +exit 0 ``` # Drawbacks @@ -205,14 +257,7 @@ mixins = [ "type=cacert:mycerts/database.crt" ] - Should stackpacks be able to define per-stack mixins? - We could support a top-level `mixins` array, and allow refinements under `[[stacks]] mixins`. If we do this, we need to distinguish between provided and required mixins (at least under `[[stacks]]`). - If buildpacks can require mixins from `bin/detect`, the stackpack could use this mechanism to require per-stack mixins. -- Should we use regex to match mixins, or should we double-down on `type=` (or similar)? - - regexs make it likely that a stackpack will be unable to distinguish packages from other types of mixins - - It will be difficult for the platform to distinguish between regex patterns and explict value to match. - - Alternatively, we could define a URN for mixins, like: - - `urn:package:libpq` - - `urn:package:build:libpq-dev` - - `urn:cacert:mycerts/database.crt` - - Naked strings, like `libpq`, could be assumed to represent packages. +- Should we use regex to match mixins? # Spec. Changes (OPTIONAL) [spec-changes]: #spec-changes From 1881dd3dead0a35b3335c99cfad893d1e2ba24b1 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Wed, 29 Jul 2020 13:02:03 -0500 Subject: [PATCH 030/140] Run all stackpacks detect by default Signed-off-by: Joe Kutner --- text/0000-app-mixins.md | 26 ++++++++++++++++---------- 1 file changed, 16 insertions(+), 10 deletions(-) diff --git a/text/0000-app-mixins.md b/text/0000-app-mixins.md index 33540684e..4739e66d0 100644 --- a/text/0000-app-mixins.md +++ b/text/0000-app-mixins.md @@ -68,15 +68,15 @@ The stackpack interface is identical to the buildpack interface (i.e. the same ` For each stackpack, the lifecycle will use [snapshotting](https://github.com/GoogleContainerTools/kaniko/blob/master/docs/designdoc.md#snapshotting-snapshotting) to capture changes made during the stackpack's build phase (excluding `/tmp`, `/cnb`, and `/layers`). Alternatively, a platform may store a new stack image to cache the changes. All of the captured changes will be included in a single layer produced as output from the stackpack. The `/layers` dir MAY NOT be used to create arbitrary layers. -A builder can use a stackpack by defining it in the `builder.toml` in the `[[stack.buildpacks]]` array of tables: +A stack can provide stackpacks by including them in the `/cnb/stackpacks` directory. By default, an implicit order will be created for all stackpacks included in a stack. The order can be overridden in the `builder.toml` with the following configuration: ``` -[[stack.buildpacks]] -id = "" -uri = "" +[[stack.order]] + [[stack.order.group]] + id = "" ``` -Each stackpack used by the builder will execute if it passes detection and it's provided mixins satisfy one of a project's required mixins. +A stackpack will only execute if it passes detection. The stackpack's snapshot layer may be modified by writing a `launch.toml` file. The `[[processes]]` and `[[slices]]` tables may be used as normal, and a new `[[excludes]]` table will be introduced with the following schema: @@ -103,7 +103,7 @@ A buildpack that can install an arbitrary list of mixins would have a `buildpack [buildpack] id = "example/apt" privileged = true -mixins = ["[^=]+"] +provides-mixins = true [[stacks]] id = "io.buildpacks.stacks.bionic" @@ -114,6 +114,14 @@ Its `bin/detect` would have the following contents: ```bash #!/usr/bin/env bash +# TODO read the mixins.toml (which has same schema as build plan) +for package in $(cat mixins.toml | yj -t | jq -r ".requires | .[] | .name"); do + cat << EOF >> $2 +[[provides]] +name = "${package}" +EOF +done + exit 0 ``` @@ -147,10 +155,9 @@ A buildpack that works on the `ubuntu-20` stack, which defines a `type=` prefix [buildpack] id = "example/cacerts" privileged = true -mixins = ["cacert"] [[stacks]] -id = "ubuntu-20" +id = "io.buildpacks.stacks.bionic" ``` Its `bin/detect` would have the following contents: @@ -190,8 +197,7 @@ The stackpack must be used with a buildpack that provides a certificate(s) for i id = "my/db-cert" [[stacks]] -id = "ubuntu-20" -mixins = ["cacert"] +id = "io.buildpacks.stacks.bionic" ``` Its `bin/detect` would require a certificate with the following contents: From 78512231d9ac2b7089ed9298ae031655d9e3cc61 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Wed, 29 Jul 2020 17:18:42 -0500 Subject: [PATCH 031/140] Update text/0000-app-mixins.md Signed-off-by: Joe Kutner Co-authored-by: Javier Romero --- text/0000-app-mixins.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-app-mixins.md b/text/0000-app-mixins.md index 4739e66d0..bbe5026bc 100644 --- a/text/0000-app-mixins.md +++ b/text/0000-app-mixins.md @@ -25,7 +25,7 @@ For that reason, this proposal defines a mechanism to implement dynamic installa - *application developer* - a person who is using buildpacks to transform their application code into an OCI image - *root buildpack* - a new type of buildpack that runs as the root user -- *stackpack* - a type of root buildpack that runs against the stack image(s) instead of an app. It is distinguished by a static list of mixins it can provide. +- *stack buildpack* - a type of root buildpack that runs against the stack image(s) instead of an app. It is distinguished by a static list of mixins it can provide. An application developer may specify a list of mixins in their application's `project.toml` file like this: From bec4c79ec2d54537bf0c077a749214f116e22289 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Wed, 29 Jul 2020 17:18:54 -0500 Subject: [PATCH 032/140] Update text/0000-app-mixins.md Signed-off-by: Joe Kutner Co-authored-by: Javier Romero --- text/0000-app-mixins.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-app-mixins.md b/text/0000-app-mixins.md index bbe5026bc..c0ce5ea0a 100644 --- a/text/0000-app-mixins.md +++ b/text/0000-app-mixins.md @@ -68,7 +68,7 @@ The stackpack interface is identical to the buildpack interface (i.e. the same ` For each stackpack, the lifecycle will use [snapshotting](https://github.com/GoogleContainerTools/kaniko/blob/master/docs/designdoc.md#snapshotting-snapshotting) to capture changes made during the stackpack's build phase (excluding `/tmp`, `/cnb`, and `/layers`). Alternatively, a platform may store a new stack image to cache the changes. All of the captured changes will be included in a single layer produced as output from the stackpack. The `/layers` dir MAY NOT be used to create arbitrary layers. -A stack can provide stackpacks by including them in the `/cnb/stackpacks` directory. By default, an implicit order will be created for all stackpacks included in a stack. The order can be overridden in the `builder.toml` with the following configuration: +A stack can provide stackpacks by including them in the `/cnb/stack/buildpacks` directory. By default, an implicit order will be created for all stackpacks included in a stack. The order can be overridden in the `builder.toml` with the following configuration: ``` [[stack.order]] From 8ee98d430e2ce55652cd53ae5ed9f344bd888889 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Wed, 29 Jul 2020 17:23:18 -0500 Subject: [PATCH 033/140] Fix apt buildpack example Signed-off-by: Joe Kutner --- text/0000-app-mixins.md | 14 +++++--------- 1 file changed, 5 insertions(+), 9 deletions(-) diff --git a/text/0000-app-mixins.md b/text/0000-app-mixins.md index c0ce5ea0a..010f34248 100644 --- a/text/0000-app-mixins.md +++ b/text/0000-app-mixins.md @@ -103,7 +103,11 @@ A buildpack that can install an arbitrary list of mixins would have a `buildpack [buildpack] id = "example/apt" privileged = true -provides-mixins = true + +[buildpack.mixins] +all = "" +names = [ "" ] + [[stacks]] id = "io.buildpacks.stacks.bionic" @@ -114,14 +118,6 @@ Its `bin/detect` would have the following contents: ```bash #!/usr/bin/env bash -# TODO read the mixins.toml (which has same schema as build plan) -for package in $(cat mixins.toml | yj -t | jq -r ".requires | .[] | .name"); do - cat << EOF >> $2 -[[provides]] -name = "${package}" -EOF -done - exit 0 ``` From 8cd657fc613769f29f84ca88673415504d57ea4c Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Wed, 29 Jul 2020 17:27:28 -0500 Subject: [PATCH 034/140] Fix buildpack.toml schema and description Signed-off-by: Joe Kutner --- text/0000-app-mixins.md | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-) diff --git a/text/0000-app-mixins.md b/text/0000-app-mixins.md index 010f34248..f4549a5cb 100644 --- a/text/0000-app-mixins.md +++ b/text/0000-app-mixins.md @@ -105,9 +105,7 @@ id = "example/apt" privileged = true [buildpack.mixins] -all = "" -names = [ "" ] - +any = true [[stacks]] id = "io.buildpacks.stacks.bionic" @@ -145,7 +143,7 @@ Support for custom CA Certificates can be accomplished with two buildpacks: a st ### CA Cert Installer Stackpack -A buildpack that works on the `ubuntu-20` stack, which defines a `type=` prefix for mixins, and installs custom CA Certificates would have a `buildpack.toml` that looks like this: +A buildpack that installs custom CA Certificates would have a `buildpack.toml` that looks like this: ```toml [buildpack] @@ -293,9 +291,16 @@ Where: [buildpack] privileged = non-idempotent = -mixins = [ "" ] + +[buildpack.mixins] +any = "" +names = [ "" ] ``` * `privileged` - when set to `true`, the lifecycle will run this buildpack as the `root` user. * `non-idempotent` - when set to `true`, indicates that the buildpack is not idempotent. The lifecycle will provide a clean filesystem from the stack image(s) before each run (i.e. no cache). -* `mixins` - a list of patterns that match mixins provided by this buildpack + +Under the `[buildpack.mixins]` table: + +* `any` - a boolean that, when true, indicates that the buildpack can provide all mixins +* `names` - a list of names that match mixins provided by this buildpack From 9c8fe9df4b74c11e3a872e62636fa05d6915fc7f Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Thu, 30 Jul 2020 12:48:22 -0500 Subject: [PATCH 035/140] Replace stackpack with stack buildpack Signed-off-by: Joe Kutner --- text/0000-app-mixins.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/text/0000-app-mixins.md b/text/0000-app-mixins.md index f4549a5cb..389287c54 100644 --- a/text/0000-app-mixins.md +++ b/text/0000-app-mixins.md @@ -51,9 +51,9 @@ When a list of mixins are required by buildpacks via the build plan and the buil 1. If at least one stackpack passes detect and provides the required mixin(s), the lifecycle will execute the stackpack build phase for all passing stackpack(s). If no stackpacks pass detect, or no stackpacks provide the required mixin, the build will fail with a message that describes the mixins that could not be provided. 1. During the lifecycle's build phase, the stackpacks that passed detection will run against the build and run images accordingly (see details below). All stackpacks will be run before the user's buildpacks. -## Stackpacks +## Stack Buildpacks - A stackpacks is a special case of buildpack that has the following properties: + A stack buildpacks (a.k.a. stackpacks) is a special case of buildpack that has the following properties: * Is run as the `root` user * Configured with `privileged = true` in the `buildpack.toml` From 5f60c74e8789e9aebd20c536baceecd58d4a5ce1 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Fri, 31 Jul 2020 10:12:12 -0500 Subject: [PATCH 036/140] Added /proc to ignored dirs Signed-off-by: Joe Kutner --- text/0000-app-mixins.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-app-mixins.md b/text/0000-app-mixins.md index 389287c54..d3cf20fda 100644 --- a/text/0000-app-mixins.md +++ b/text/0000-app-mixins.md @@ -66,7 +66,7 @@ When a list of mixins are required by buildpacks via the build plan and the buil The stackpack interface is identical to the buildpack interface (i.e. the same `bin/detect` and `bin/build` scripts are required). However, some of the context it is run in is different from regular buildpacks. -For each stackpack, the lifecycle will use [snapshotting](https://github.com/GoogleContainerTools/kaniko/blob/master/docs/designdoc.md#snapshotting-snapshotting) to capture changes made during the stackpack's build phase (excluding `/tmp`, `/cnb`, and `/layers`). Alternatively, a platform may store a new stack image to cache the changes. All of the captured changes will be included in a single layer produced as output from the stackpack. The `/layers` dir MAY NOT be used to create arbitrary layers. +For each stackpack, the lifecycle will use [snapshotting](https://github.com/GoogleContainerTools/kaniko/blob/master/docs/designdoc.md#snapshotting-snapshotting) to capture changes made during the stackpack's build phase (excluding `/tmp`, `/proc`, `/cnb`, and `/layers`). Alternatively, a platform may store a new stack image to cache the changes. All of the captured changes will be included in a single layer produced as output from the stackpack. The `/layers` dir MAY NOT be used to create arbitrary layers. A stack can provide stackpacks by including them in the `/cnb/stack/buildpacks` directory. By default, an implicit order will be created for all stackpacks included in a stack. The order can be overridden in the `builder.toml` with the following configuration: From efca813f3307fc122ae73aa874ff4fd70e75bdb9 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Fri, 7 Aug 2020 14:56:56 -0500 Subject: [PATCH 037/140] Updates based on 8/6 WG discussion Signed-off-by: Joe Kutner --- text/0000-app-mixins.md | 37 ++++++++++++++++++++++++++++++------- 1 file changed, 30 insertions(+), 7 deletions(-) diff --git a/text/0000-app-mixins.md b/text/0000-app-mixins.md index d3cf20fda..638e4425b 100644 --- a/text/0000-app-mixins.md +++ b/text/0000-app-mixins.md @@ -66,7 +66,20 @@ When a list of mixins are required by buildpacks via the build plan and the buil The stackpack interface is identical to the buildpack interface (i.e. the same `bin/detect` and `bin/build` scripts are required). However, some of the context it is run in is different from regular buildpacks. -For each stackpack, the lifecycle will use [snapshotting](https://github.com/GoogleContainerTools/kaniko/blob/master/docs/designdoc.md#snapshotting-snapshotting) to capture changes made during the stackpack's build phase (excluding `/tmp`, `/proc`, `/cnb`, and `/layers`). Alternatively, a platform may store a new stack image to cache the changes. All of the captured changes will be included in a single layer produced as output from the stackpack. The `/layers` dir MAY NOT be used to create arbitrary layers. +For each stackpack, the lifecycle will use [snapshotting](https://github.com/GoogleContainerTools/kaniko/blob/master/docs/designdoc.md#snapshotting-snapshotting) to capture changes made during the stackpack's build phase excluding the following directories: + +* `/tmp` +* `/cnb` +* `/layers` +* `/workspace` +* `/dev` +* `/sys` +* `/proc` +* `/var/run/secrets` +* `/etc/hostname`, `/etc/hosts`, `/etc/mtab`, `/etc/resolv.conf` +* `/.dockerenv` + +Alternatively, a platform may store a new stack image to cache the changes. All of the captured changes will be included in a single layer produced as output from the stackpack. The `/layers` dir MAY NOT be used to create arbitrary layers. A stack can provide stackpacks by including them in the `/cnb/stack/buildpacks` directory. By default, an implicit order will be created for all stackpacks included in a stack. The order can be overridden in the `builder.toml` with the following configuration: @@ -76,18 +89,27 @@ A stack can provide stackpacks by including them in the `/cnb/stack/buildpacks` id = "" ``` -A stackpack will only execute if it passes detection. +A stackpack will only execute if it passes detection. When the stackpack is executed, it's detect and build scripts use the same parameters as the regular buildpacks. But the arguments may differ. FOr example, the first positional argument to the `bin/build` (the `` directory) MUST NOT be a standard layers directory adhering to the [Buildpacks Build specification](https://github.com/buildpacks/spec/blob/main/buildpack.md#build). -The stackpack's snapshot layer may be modified by writing a `launch.toml` file. The `[[processes]]` and `[[slices]]` tables may be used as normal, and a new `[[excludes]]` table will be introduced with the following schema: +The stackpack's snapshot layer may be modified by writing a `stack.toml` file with the following schema: ``` -[[excludes]] +[[build.restores]] +paths = [""] + +[[run.excludes]] paths = [""] cache = false ``` Where: +`[[build.restores]]` defines the directories and files that will be restored at build time even when the base image is modified: + +* `paths` = a list of paths to always restore + +`[[run.excludes]]` defines the directories and files that will be excluded from the launch image: + * `paths` = a list of paths to exclude from the layer * `cache` = if true, the paths will be excluded from the launch image layer, but will be included in the cache layer. @@ -104,11 +126,12 @@ A buildpack that can install an arbitrary list of mixins would have a `buildpack id = "example/apt" privileged = true -[buildpack.mixins] -any = true - [[stacks]] id = "io.buildpacks.stacks.bionic" +mixins = [ "set=cnb-essentials" ] + + [stacks.extends] + mixins = [ "*" ] ``` Its `bin/detect` would have the following contents: From dcb29c990939b4acedc0fcea5a30a14f1f6b4836 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Wed, 12 Aug 2020 09:08:17 -0500 Subject: [PATCH 038/140] Move schema changes to the end of the RFC Signed-off-by: Joe Kutner --- text/0000-app-mixins.md | 46 ++++++++++++++++++++++------------------- 1 file changed, 25 insertions(+), 21 deletions(-) diff --git a/text/0000-app-mixins.md b/text/0000-app-mixins.md index 638e4425b..e28039608 100644 --- a/text/0000-app-mixins.md +++ b/text/0000-app-mixins.md @@ -91,27 +91,7 @@ A stack can provide stackpacks by including them in the `/cnb/stack/buildpacks` A stackpack will only execute if it passes detection. When the stackpack is executed, it's detect and build scripts use the same parameters as the regular buildpacks. But the arguments may differ. FOr example, the first positional argument to the `bin/build` (the `` directory) MUST NOT be a standard layers directory adhering to the [Buildpacks Build specification](https://github.com/buildpacks/spec/blob/main/buildpack.md#build). -The stackpack's snapshot layer may be modified by writing a `stack.toml` file with the following schema: - -``` -[[build.restores]] -paths = [""] - -[[run.excludes]] -paths = [""] -cache = false -``` - -Where: - -`[[build.restores]]` defines the directories and files that will be restored at build time even when the base image is modified: - -* `paths` = a list of paths to always restore - -`[[run.excludes]]` defines the directories and files that will be excluded from the launch image: - -* `paths` = a list of paths to exclude from the layer -* `cache` = if true, the paths will be excluded from the launch image layer, but will be included in the cache layer. +The stackpack's snapshot layer may be modified by writing a `stack.toml` file. The `stack.toml` will define paths that will be restored even when the base image changes (ex. package indicies) and paths that will be excluded from the launch image (ex. `/var/cache`). ## Rebasing an App @@ -327,3 +307,27 @@ Under the `[buildpack.mixins]` table: * `any` - a boolean that, when true, indicates that the buildpack can provide all mixins * `names` - a list of names that match mixins provided by this buildpack + +## stack.toml (TOML) + +The stackpack's snapshot layer may be modified by writing a `stack.toml` file with the following schema: + +``` +[[build.restores]] +paths = [""] + +[[run.excludes]] +paths = [""] +cache = false +``` + +Where: + +`[[build.restores]]` defines the directories and files that will be restored at build time even when the base image is modified: + +* `paths` = a list of paths to always restore + +`[[run.excludes]]` defines the directories and files that will be excluded from the launch image: + +* `paths` = a list of paths to exclude from the layer +* `cache` = if true, the paths will be excluded from the launch image layer, but will be included in the cache layer. \ No newline at end of file From c19d73355e6fc82eed45db9c51fe6dbdc2976451 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Wed, 12 Aug 2020 13:48:05 -0500 Subject: [PATCH 039/140] Move stackpack excluded dirs to spec section Signed-off-by: Joe Kutner --- text/0000-app-mixins.md | 32 +++++++++++++++++--------------- 1 file changed, 17 insertions(+), 15 deletions(-) diff --git a/text/0000-app-mixins.md b/text/0000-app-mixins.md index e28039608..0f59f5dbf 100644 --- a/text/0000-app-mixins.md +++ b/text/0000-app-mixins.md @@ -66,18 +66,7 @@ When a list of mixins are required by buildpacks via the build plan and the buil The stackpack interface is identical to the buildpack interface (i.e. the same `bin/detect` and `bin/build` scripts are required). However, some of the context it is run in is different from regular buildpacks. -For each stackpack, the lifecycle will use [snapshotting](https://github.com/GoogleContainerTools/kaniko/blob/master/docs/designdoc.md#snapshotting-snapshotting) to capture changes made during the stackpack's build phase excluding the following directories: - -* `/tmp` -* `/cnb` -* `/layers` -* `/workspace` -* `/dev` -* `/sys` -* `/proc` -* `/var/run/secrets` -* `/etc/hostname`, `/etc/hosts`, `/etc/mtab`, `/etc/resolv.conf` -* `/.dockerenv` +For each stackpack, the lifecycle will use [snapshotting](https://github.com/GoogleContainerTools/kaniko/blob/master/docs/designdoc.md#snapshotting-snapshotting) to capture changes made during the stackpack's build phase excluding the a few specific directories and files. Alternatively, a platform may store a new stack image to cache the changes. All of the captured changes will be included in a single layer produced as output from the stackpack. The `/layers` dir MAY NOT be used to create arbitrary layers. @@ -265,13 +254,26 @@ exit 0 # Spec. Changes (OPTIONAL) [spec-changes]: #spec-changes -## Stackpacks +A number of changes to the Platform Specification will be required to execute Stack Buildpacks. Those changes will be defined in a separate RFC. + +## Stack buildpacks -Stackpacks are identical to other buildpacks, with the following exceptions: +Stack buildpacks are identical to other buildpacks, with the following exceptions: 1. The `` directory is NOT writable. 1. The working directory WILL NOT contain application source code during the build phase. -1. All changes made to the filesystem (with the exception of `/tmp`) during the execution of the stackpack's `bin/build` will be snapshotted and stored as a single layer. +1. All changes made to the filesystem during the execution of the stackpack's `bin/build` will be snapshotted and stored as a single layer, with the exception of the following directories: + +* `/tmp` +* `/cnb` +* `/layers` +* `/workspace` +* `/dev` +* `/sys` +* `/proc` +* `/var/run/secrets` +* `/etc/hostname`, `/etc/hosts`, `/etc/mtab`, `/etc/resolv.conf` +* `/.dockerenv` ## launch.toml (TOML) From 2fed8647cfff9790176e8fccb2a32ef4168af4ce Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Wed, 12 Aug 2020 14:40:00 -0500 Subject: [PATCH 040/140] Update text/0000-app-mixins.md Signed-off-by: Joe Kutner Co-authored-by: Jesse Brown --- text/0000-app-mixins.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/text/0000-app-mixins.md b/text/0000-app-mixins.md index 0f59f5dbf..9228ba6b4 100644 --- a/text/0000-app-mixins.md +++ b/text/0000-app-mixins.md @@ -78,7 +78,7 @@ A stack can provide stackpacks by including them in the `/cnb/stack/buildpacks` id = "" ``` -A stackpack will only execute if it passes detection. When the stackpack is executed, it's detect and build scripts use the same parameters as the regular buildpacks. But the arguments may differ. FOr example, the first positional argument to the `bin/build` (the `` directory) MUST NOT be a standard layers directory adhering to the [Buildpacks Build specification](https://github.com/buildpacks/spec/blob/main/buildpack.md#build). +A stackpack will only execute if it passes detection. When the stackpack is executed, it's detect and build scripts use the same parameters as the regular buildpacks. But the arguments may differ. For example, the first positional argument to the `bin/build` (the `` directory) MUST NOT be a standard layers directory adhering to the [Buildpacks Build specification](https://github.com/buildpacks/spec/blob/main/buildpack.md#build). The stackpack's snapshot layer may be modified by writing a `stack.toml` file. The `stack.toml` will define paths that will be restored even when the base image changes (ex. package indicies) and paths that will be excluded from the launch image (ex. `/var/cache`). @@ -332,4 +332,4 @@ Where: `[[run.excludes]]` defines the directories and files that will be excluded from the launch image: * `paths` = a list of paths to exclude from the layer -* `cache` = if true, the paths will be excluded from the launch image layer, but will be included in the cache layer. \ No newline at end of file +* `cache` = if true, the paths will be excluded from the launch image layer, but will be included in the cache layer. From 57bdc7b92fe15e1acdf599b333180432f776f00e Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Wed, 12 Aug 2020 14:41:42 -0500 Subject: [PATCH 041/140] Rename RFC for stack buildpacks Signed-off-by: Joe Kutner --- .../{0000-app-mixins.md => 0000-app-mixins-stack-buildpacks.md} | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) rename text/{0000-app-mixins.md => 0000-app-mixins-stack-buildpacks.md} (99%) diff --git a/text/0000-app-mixins.md b/text/0000-app-mixins-stack-buildpacks.md similarity index 99% rename from text/0000-app-mixins.md rename to text/0000-app-mixins-stack-buildpacks.md index 9228ba6b4..d6e04d873 100644 --- a/text/0000-app-mixins.md +++ b/text/0000-app-mixins-stack-buildpacks.md @@ -1,6 +1,6 @@ # Meta [meta]: #meta -- Name: Application Mixins +- Name: Application Mixins and Stack Buildpacks - Start Date: 2020-06-22 - Author(s): [Joe Kutner](https://github.com/jkutner/) - RFC Pull Request: (leave blank) From 38625f646c3e4cc25de58b8de254ff2a7c80c447 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Thu, 27 Aug 2020 12:31:55 -0500 Subject: [PATCH 042/140] Divide app-mixins RFC into two RFCs. this is the stack buildpack RFC Signed-off-by: Joe Kutner --- ...buildpacks.md => 0000-stack-buildpacks.md} | 71 +++++-------------- 1 file changed, 16 insertions(+), 55 deletions(-) rename text/{0000-app-mixins-stack-buildpacks.md => 0000-stack-buildpacks.md} (66%) diff --git a/text/0000-app-mixins-stack-buildpacks.md b/text/0000-stack-buildpacks.md similarity index 66% rename from text/0000-app-mixins-stack-buildpacks.md rename to text/0000-stack-buildpacks.md index d6e04d873..75fd4309c 100644 --- a/text/0000-app-mixins-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -1,8 +1,8 @@ # Meta [meta]: #meta -- Name: Application Mixins and Stack Buildpacks -- Start Date: 2020-06-22 -- Author(s): [Joe Kutner](https://github.com/jkutner/) +- Name: Stack Buildpacks +- Start Date: 2020-08-27 +- Author(s): [@jkutner](@jkutner) - RFC Pull Request: (leave blank) - CNB Pull Request: (leave blank) - CNB Issue: (leave blank) @@ -11,49 +11,35 @@ # Summary [summary]: #summary -This is a proposal for allowing application developers (buildpack users) to specify mixins that will be dynamically included with their build and run images. +This is proposal for a new type of buildpack that runs against a stack in order to extend it in ways that are only possible by running privileged commands. # Motivation [motivation]: #motivation -Mixins already allow buildpack authors to create buildpacks that depend on an extended set of OS packages without affecting build time, but it's common for application code to depend on OS packages too. But allowing buildpacks to arbitrarily install OS packages during build would drastically increase build time, especially when CNB tooling is used to build or rebase many apps with similar package requirements. +Normal buildpacks do not run as root. This is an intentional design decision that ensures operations like `rebase` will work on day-2. -For that reason, this proposal defines a mechanism to implement dynamic installation of mixins at build time while minimizing the performance impact. This is accomplished by allowing stack images to satisfy mixin requires statically and/or reject dynamic mixin installation early in the build process. +However, many applications and buildpacks require modifications to the stack they run on, such as adding system packages or custom certificates. For this reason, we need a mechanism that buildpack authors and buildpack users and leverage to extend their stacks. # What it is [what-it-is]: #what-it-is -- *application developer* - a person who is using buildpacks to transform their application code into an OCI image - *root buildpack* - a new type of buildpack that runs as the root user - *stack buildpack* - a type of root buildpack that runs against the stack image(s) instead of an app. It is distinguished by a static list of mixins it can provide. +- *userspace buildpack* - the traditional definition of a buildpack (i.e. does not run as root, and runs against an app) -An application developer may specify a list of mixins in their application's `project.toml` file like this: +A new type of buildpack, called a Stack buildpack, may run against a stack (both build and run images) in order to extend it in ways that are only possible by running privileged commands. -```toml -[build] -mixins = [ "" ] -``` - -When a command like `pack build` is run, the list of mixins will be processed before buildpacks are run. For each mixin name, the following will happen: +A stack provider may choose to include stack buildpacks with the stack they distribute. If a stack includes any stack buildpacks, the following will occur when the build phase starts: -* If the mixin name is prefixed with `build:` (as per [RFC-0006](https://github.com/buildpacks/rfcs/blob/main/text/0006-stage-specific-mixins.md)) the mixin will be dynamically added to the build image. -* If the mixin name is prefixed with `run:` (as per [RFC-0006](https://github.com/buildpacks/rfcs/blob/main/text/0006-stage-specific-mixins.md)) the mixin will be dynamically added to the run image. -* If the mixin name does not have a prefix it will be dynamically added to both the build and run stack images. +1. The lifecycle will run the detect phase for all stackpacks defined in the builder. +1. The lifecycle will execute the stack buildpack build phase for all passing stackpack(s). If no stack buildpacks pass detect, the build will continue the build phase as normal (running userspace buildpacks). +1. After the lifecycle's build phase, the lifecycle will begin a new phase called _extend_. +1. During the extend phase, stack buildpacks that passed detection will run against the run images accordingly (see details below). # How it Works [how-it-works]: #how-it-works -When a list of mixins are required by buildpacks via the build plan and the build phase starts: - -1. The lifecycle will compare the list of mixins to those provided by the stack. If all mixin names are provided by the stack, no further action is required. -1. If any requested mixin is not provided by the stack, the lifecycle will run the detect phase for all stackpacks defined in the builder. -1. The lifecycle will compare the mixins added to the build plan to see if they match they required mixins. -1. If at least one stackpack passes detect and provides the required mixin(s), the lifecycle will execute the stackpack build phase for all passing stackpack(s). If no stackpacks pass detect, or no stackpacks provide the required mixin, the build will fail with a message that describes the mixins that could not be provided. -1. During the lifecycle's build phase, the stackpacks that passed detection will run against the build and run images accordingly (see details below). All stackpacks will be run before the user's buildpacks. - -## Stack Buildpacks - - A stack buildpacks (a.k.a. stackpacks) is a special case of buildpack that has the following properties: + A stack buildpack (a.k.a. stackpack) is a special case of buildpack that has the following properties: * Is run as the `root` user * Configured with `privileged = true` in the `buildpack.toml` @@ -78,7 +64,7 @@ A stack can provide stackpacks by including them in the `/cnb/stack/buildpacks` id = "" ``` -A stackpack will only execute if it passes detection. When the stackpack is executed, it's detect and build scripts use the same parameters as the regular buildpacks. But the arguments may differ. For example, the first positional argument to the `bin/build` (the `` directory) MUST NOT be a standard layers directory adhering to the [Buildpacks Build specification](https://github.com/buildpacks/spec/blob/main/buildpack.md#build). +A stackpack will only execute if it passes detection. When the stackpack is executed, its detect and build scripts use the same parameters as the regular buildpacks. But the arguments may differ. For example, the first positional argument to the `bin/build` (the `` directory) MUST NOT be a standard layers directory adhering to the [Buildpacks Build specification](https://github.com/buildpacks/spec/blob/main/buildpack.md#build). The stackpack's snapshot layer may be modified by writing a `stack.toml` file. The `stack.toml` will define paths that will be restored even when the base image changes (ex. package indicies) and paths that will be excluded from the launch image (ex. `/var/cache`). @@ -225,7 +211,7 @@ exit 0 # Drawbacks [drawbacks]: #drawbacks -- Mixins are less flexible than a generic [root buildpack](https://github.com/buildpacks/rfcs/blob/app-image-ext-buildpacks/text/0000-app-image-extensions.md). +- Stack buildpacks cannot be provided by the end user (the app developer), they can only by provided by stack creators # Alternatives [alternatives]: #alternatives @@ -237,14 +223,12 @@ exit 0 # Prior Art [prior-art]: #prior-art -- [RFC-0006: Stage specific mixins](https://github.com/buildpacks/rfcs/blob/main/text/0006-stage-specific-mixins.md) - The term "non-idempotent" is used in section 9.1.2 of [Hypertext Transfer Protocol -- HTTP/1.1 (RFC 2616)](https://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html) # Unresolved Questions [unresolved-questions]: #unresolved-questions - Should the stackpack's detect have read-only access to the app? - - Does a stackpack even need a detect phase? - This would likely be driven by a stackpack that does not provide mixins, but instead dynamically contributes to the build plan based on the contents of the app source code. I don't know if we have a use case for this, but I can imaging a buildpack that reads environment variables as input to some function. - Should stackpacks be able to define per-stack mixins? - We could support a top-level `mixins` array, and allow refinements under `[[stacks]] mixins`. If we do this, we need to distinguish between provided and required mixins (at least under `[[stacks]]`). @@ -310,26 +294,3 @@ Under the `[buildpack.mixins]` table: * `any` - a boolean that, when true, indicates that the buildpack can provide all mixins * `names` - a list of names that match mixins provided by this buildpack -## stack.toml (TOML) - -The stackpack's snapshot layer may be modified by writing a `stack.toml` file with the following schema: - -``` -[[build.restores]] -paths = [""] - -[[run.excludes]] -paths = [""] -cache = false -``` - -Where: - -`[[build.restores]]` defines the directories and files that will be restored at build time even when the base image is modified: - -* `paths` = a list of paths to always restore - -`[[run.excludes]]` defines the directories and files that will be excluded from the launch image: - -* `paths` = a list of paths to exclude from the layer -* `cache` = if true, the paths will be excluded from the launch image layer, but will be included in the cache layer. From c16c9f3888309bf7f49490b0570dbead837a2de1 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Thu, 27 Aug 2020 13:32:40 -0500 Subject: [PATCH 043/140] Added details about mixins to stackpacks Signed-off-by: Joe Kutner --- text/0000-stack-buildpacks.md | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index 75fd4309c..f84374052 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -16,7 +16,7 @@ This is proposal for a new type of buildpack that runs against a stack in order # Motivation [motivation]: #motivation -Normal buildpacks do not run as root. This is an intentional design decision that ensures operations like `rebase` will work on day-2. +Normally, buildpacks do not run as root. This is an intentional design decision that ensures operations like `rebase` will work on day-2. However, many applications and buildpacks require modifications to the stack they run on, such as adding system packages or custom certificates. For this reason, we need a mechanism that buildpack authors and buildpack users and leverage to extend their stacks. @@ -27,15 +27,17 @@ However, many applications and buildpacks require modifications to the stack the - *stack buildpack* - a type of root buildpack that runs against the stack image(s) instead of an app. It is distinguished by a static list of mixins it can provide. - *userspace buildpack* - the traditional definition of a buildpack (i.e. does not run as root, and runs against an app) -A new type of buildpack, called a Stack buildpack, may run against a stack (both build and run images) in order to extend it in ways that are only possible by running privileged commands. +A new type of buildpack, called a Stack buildpack, may run against a stack (both build and run images) in order to extend it in ways that are only possible by running privileged commands. A stackpack may also define a list of mixins that it provides to the stack. In this way, a stack that is missing a mixin required by a buildpack may have that mixin provided by a stack buildpack. A stack provider may choose to include stack buildpacks with the stack they distribute. If a stack includes any stack buildpacks, the following will occur when the build phase starts: +1. If any requested mixin is not provided by the stack, the lifecycle will compare the missing mixins to the static list of mixins provided by stack buildpacks. If any mixins are still not provided, the build will fail. 1. The lifecycle will run the detect phase for all stackpacks defined in the builder. 1. The lifecycle will execute the stack buildpack build phase for all passing stackpack(s). If no stack buildpacks pass detect, the build will continue the build phase as normal (running userspace buildpacks). 1. After the lifecycle's build phase, the lifecycle will begin a new phase called _extend_. 1. During the extend phase, stack buildpacks that passed detection will run against the run images accordingly (see details below). + # How it Works [how-it-works]: #how-it-works @@ -74,6 +76,8 @@ Before a launch image is rebased, the platform must re-run the any stackpacks th ## Example: Apt Buildpack +(**note**: this is only an example. A real Apt Buildpack would probably be more robust). + A buildpack that can install an arbitrary list of mixins would have a `buildpack.toml` like this: ```toml @@ -81,12 +85,11 @@ A buildpack that can install an arbitrary list of mixins would have a `buildpack id = "example/apt" privileged = true +[buildpack.mixins] +any = true + [[stacks]] id = "io.buildpacks.stacks.bionic" -mixins = [ "set=cnb-essentials" ] - - [stacks.extends] - mixins = [ "*" ] ``` Its `bin/detect` would have the following contents: @@ -228,12 +231,12 @@ exit 0 # Unresolved Questions [unresolved-questions]: #unresolved-questions +- Do we want (or need) a `CNB_STACK_TYPE` env var so that a stack buildpack can behave differently on each part of the stack? - Should the stackpack's detect have read-only access to the app? - This would likely be driven by a stackpack that does not provide mixins, but instead dynamically contributes to the build plan based on the contents of the app source code. I don't know if we have a use case for this, but I can imaging a buildpack that reads environment variables as input to some function. - Should stackpacks be able to define per-stack mixins? - We could support a top-level `mixins` array, and allow refinements under `[[stacks]] mixins`. If we do this, we need to distinguish between provided and required mixins (at least under `[[stacks]]`). - - If buildpacks can require mixins from `bin/detect`, the stackpack could use this mechanism to require per-stack mixins. -- Should we use regex to match mixins? + - If userspace buildpacks can require mixins from `bin/detect`, the stackpack could use this mechanism to require per-stack mixins. # Spec. Changes (OPTIONAL) [spec-changes]: #spec-changes From aae57b21066b2362c906586055c8bd2dd93d6fdf Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Tue, 8 Sep 2020 08:51:06 -0500 Subject: [PATCH 044/140] Updates based on PR feedback Signed-off-by: Joe Kutner --- text/0000-stack-buildpacks.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index f84374052..ffdd09d5f 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -27,14 +27,14 @@ However, many applications and buildpacks require modifications to the stack the - *stack buildpack* - a type of root buildpack that runs against the stack image(s) instead of an app. It is distinguished by a static list of mixins it can provide. - *userspace buildpack* - the traditional definition of a buildpack (i.e. does not run as root, and runs against an app) -A new type of buildpack, called a Stack buildpack, may run against a stack (both build and run images) in order to extend it in ways that are only possible by running privileged commands. A stackpack may also define a list of mixins that it provides to the stack. In this way, a stack that is missing a mixin required by a buildpack may have that mixin provided by a stack buildpack. +A new type of buildpack, called a Stack buildpack, may run against a stack (both build and run images) in order to extend it in ways that are only possible by running privileged commands. A stackpack may also define a list of mixins that it provides to the stack, or indicate that it will provide _any_ mixin. In this way, a stack that is missing a mixin required by a buildpack may have that mixin provided by a stack buildpack. A stack provider may choose to include stack buildpacks with the stack they distribute. If a stack includes any stack buildpacks, the following will occur when the build phase starts: 1. If any requested mixin is not provided by the stack, the lifecycle will compare the missing mixins to the static list of mixins provided by stack buildpacks. If any mixins are still not provided, the build will fail. 1. The lifecycle will run the detect phase for all stackpacks defined in the builder. 1. The lifecycle will execute the stack buildpack build phase for all passing stackpack(s). If no stack buildpacks pass detect, the build will continue the build phase as normal (running userspace buildpacks). -1. After the lifecycle's build phase, the lifecycle will begin a new phase called _extend_. +1. After, during, or before the lifecycle's build phase, the lifecycle will begin a new phase called _extend_. 1. During the extend phase, stack buildpacks that passed detection will run against the run images accordingly (see details below). @@ -50,13 +50,13 @@ A stack provider may choose to include stack buildpacks with the stack they dist * Is run before all regular buildpacks * Is run against both the build and run images * Is distributed with the stack run and/or build image -* May not write to the `/layers` directory +* May not create layers using the `` directory The stackpack interface is identical to the buildpack interface (i.e. the same `bin/detect` and `bin/build` scripts are required). However, some of the context it is run in is different from regular buildpacks. For each stackpack, the lifecycle will use [snapshotting](https://github.com/GoogleContainerTools/kaniko/blob/master/docs/designdoc.md#snapshotting-snapshotting) to capture changes made during the stackpack's build phase excluding the a few specific directories and files. -Alternatively, a platform may store a new stack image to cache the changes. All of the captured changes will be included in a single layer produced as output from the stackpack. The `/layers` dir MAY NOT be used to create arbitrary layers. +Additionally, a platform may store a new stack image to cache the changes. All of the captured changes will be included in a single layer produced as output from the stackpack. The `/layers` dir MAY NOT be used to create arbitrary layers. A stack can provide stackpacks by including them in the `/cnb/stack/buildpacks` directory. By default, an implicit order will be created for all stackpacks included in a stack. The order can be overridden in the `builder.toml` with the following configuration: @@ -247,7 +247,7 @@ A number of changes to the Platform Specification will be required to execute St Stack buildpacks are identical to other buildpacks, with the following exceptions: -1. The `` directory is NOT writable. +1. The `` directory WILL NOT be used to create layers. 1. The working directory WILL NOT contain application source code during the build phase. 1. All changes made to the filesystem during the execution of the stackpack's `bin/build` will be snapshotted and stored as a single layer, with the exception of the following directories: From c4d2fbc3a2c3f9e6ceada0074ab6e967a5416a62 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Fri, 11 Sep 2020 10:03:59 -0500 Subject: [PATCH 045/140] Update text/0000-stack-buildpacks.md Signed-off-by: Joe Kutner Co-authored-by: Emily Casey --- text/0000-stack-buildpacks.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index ffdd09d5f..132d6cea7 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -24,7 +24,7 @@ However, many applications and buildpacks require modifications to the stack the [what-it-is]: #what-it-is - *root buildpack* - a new type of buildpack that runs as the root user -- *stack buildpack* - a type of root buildpack that runs against the stack image(s) instead of an app. It is distinguished by a static list of mixins it can provide. +- *stack buildpack* - a type of root buildpack that runs against the stack image(s) instead of an app. Stack buildpacks must not make changes to the build and run images that either violate stack [compatibility guarantees](https://github.com/buildpacks/spec/blob/main/platform.md#compatibility-guarantees) or violate the contract defined by that stack's author. - *userspace buildpack* - the traditional definition of a buildpack (i.e. does not run as root, and runs against an app) A new type of buildpack, called a Stack buildpack, may run against a stack (both build and run images) in order to extend it in ways that are only possible by running privileged commands. A stackpack may also define a list of mixins that it provides to the stack, or indicate that it will provide _any_ mixin. In this way, a stack that is missing a mixin required by a buildpack may have that mixin provided by a stack buildpack. @@ -296,4 +296,3 @@ Under the `[buildpack.mixins]` table: * `any` - a boolean that, when true, indicates that the buildpack can provide all mixins * `names` - a list of names that match mixins provided by this buildpack - From f9fa626e0c119f23178e5e531b56cf144123d5d2 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Fri, 11 Sep 2020 10:11:34 -0500 Subject: [PATCH 046/140] Added mixins to build plan to support stackpacks Signed-off-by: Joe Kutner --- text/0000-stack-buildpacks.md | 82 +++++++++++++++++++++++++++++++++-- 1 file changed, 79 insertions(+), 3 deletions(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index 132d6cea7..6c6b0c604 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -70,6 +70,40 @@ A stackpack will only execute if it passes detection. When the stackpack is exec The stackpack's snapshot layer may be modified by writing a `stack.toml` file. The `stack.toml` will define paths that will be restored even when the base image changes (ex. package indicies) and paths that will be excluded from the launch image (ex. `/var/cache`). +## Providing Mixins + +A stack buildpack MAY define a set of mixins it provides in either of two ways: + +1. Statically in the `buildpack.toml` +1. Dynamically in the Build Plan. + +A stack buildpack MAY define a set of stack specific mixins in the `buildpack.toml` with the following schema: + +```toml +[[buildpack.provides.stacks]] +id = "" +any = +mixins = [ "mixin name" ] +``` + +Additionally, mixins MAY be dyncamically provided in the build plan: + +``` +[[provides]] +name = "" +mixin = +``` + +A userspace buildpack MAY require mixins in the build plan + +``` +[[requires]] +name = "" +mixin = +``` + +A userspace buildpack MAY NOT provide mixins in the build plan. + ## Rebasing an App Before a launch image is rebased, the platform must re-run the any stackpacks that were used to build the launch image against the new run-image. Then, the rebase operation can be performed as normal, while including the stackpack layers as part of the stack. This will be made possible by including the stackpack in the run-image, but because the stackpack detect phase is not run, the operation does not need access to the application source. @@ -211,6 +245,19 @@ Its `bin/build` would do nothing, and `have the following contents: exit 0 ``` +## Example: jq mixin + +A buildpack may that requires the `jq` package may have it provided by either a stack, stack buildpack, or userspace buildpack. + +```toml +[[or.requires]] +name = "jq" +mixin = true + +[[or.requires]] +name = "jq" +``` + # Drawbacks [drawbacks]: #drawbacks @@ -284,9 +331,10 @@ Where: privileged = non-idempotent = -[buildpack.mixins] -any = "" -names = [ "" ] +[[buildpack.provides.stacks]] +id = "" +any = +mixins = [ "" ] ``` * `privileged` - when set to `true`, the lifecycle will run this buildpack as the `root` user. @@ -296,3 +344,31 @@ Under the `[buildpack.mixins]` table: * `any` - a boolean that, when true, indicates that the buildpack can provide all mixins * `names` - a list of names that match mixins provided by this buildpack + +## Build Plan (TOML) + +``` +[[provides]] +name = "" +mixin = + +[[requires]] +name = "" +mixin = + +[requires.metadata] +# buildpack-specific data + +[[or]] + +[[or.provides]] +name = "" +mixin = + +[[or.requires]] +name = "" +mixin = + +[or.requires.metadata] +# buildpack-specific data +``` From d7d3ff30f96bc6088513ce527310fa69788bbef7 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Fri, 11 Sep 2020 14:58:07 -0500 Subject: [PATCH 047/140] Fix path to launch.toml in stackpack RFC Signed-off-by: Joe Kutner --- text/0000-stack-buildpacks.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index 6c6b0c604..be2112764 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -145,7 +145,7 @@ for package in $(cat $3 | yj -t | jq -r ".entries | .[] | .name"); do apt install $package done -cat << EOF > launch.toml +cat << EOF > $1/launch.toml [[excludes]] paths = [ "/var/cache" ] cache = true From 0e22d03b9169cb0657db91cb9715cc7a10822b1a Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Fri, 11 Sep 2020 14:58:59 -0500 Subject: [PATCH 048/140] Remove mentions of stack.toml Signed-off-by: Joe Kutner --- text/0000-stack-buildpacks.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index be2112764..08556b127 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -68,7 +68,7 @@ A stack can provide stackpacks by including them in the `/cnb/stack/buildpacks` A stackpack will only execute if it passes detection. When the stackpack is executed, its detect and build scripts use the same parameters as the regular buildpacks. But the arguments may differ. For example, the first positional argument to the `bin/build` (the `` directory) MUST NOT be a standard layers directory adhering to the [Buildpacks Build specification](https://github.com/buildpacks/spec/blob/main/buildpack.md#build). -The stackpack's snapshot layer may be modified by writing a `stack.toml` file. The `stack.toml` will define paths that will be restored even when the base image changes (ex. package indicies) and paths that will be excluded from the launch image (ex. `/var/cache`). +The stackpack's snapshot layer may be enriched by writing a `launch.toml` file. The `launch.toml` will define paths that will be restored even when the base image changes (ex. package indicies) and paths that will be excluded from the launch image (ex. `/var/cache`). ## Providing Mixins From 3cba356d047d5eda1ca0e425a3996a94c8aaaea4 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Fri, 11 Sep 2020 16:19:06 -0500 Subject: [PATCH 049/140] Updated 'what it is' section of stack buildpacks based on feedback Signed-off-by: Joe Kutner --- text/0000-stack-buildpacks.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index 08556b127..4efa762c2 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -27,11 +27,13 @@ However, many applications and buildpacks require modifications to the stack the - *stack buildpack* - a type of root buildpack that runs against the stack image(s) instead of an app. Stack buildpacks must not make changes to the build and run images that either violate stack [compatibility guarantees](https://github.com/buildpacks/spec/blob/main/platform.md#compatibility-guarantees) or violate the contract defined by that stack's author. - *userspace buildpack* - the traditional definition of a buildpack (i.e. does not run as root, and runs against an app) -A new type of buildpack, called a Stack buildpack, may run against a stack (both build and run images) in order to extend it in ways that are only possible by running privileged commands. A stackpack may also define a list of mixins that it provides to the stack, or indicate that it will provide _any_ mixin. In this way, a stack that is missing a mixin required by a buildpack may have that mixin provided by a stack buildpack. +A new type of buildpack, called a Stack buildpack, may run against a stack (both build and run images) in order to extend it in ways that are only possible by running privileged commands. Unlike userspace buildpacks, stack buildpack can modify any path on the filesystem. Userspace buildpack can only create/modify disjoint layers (either by adding a dir to `` or modifying an app slice), which makes possible features like individual layer reuse that is independent or ordering. + +A stackpack may also define a list of mixins that it provides to the stack, or indicate that it will provide _any_ mixin. In this way, a stack that is missing a mixin required by a buildpack may have that mixin provided by a stack buildpack. A stack provider may choose to include stack buildpacks with the stack they distribute. If a stack includes any stack buildpacks, the following will occur when the build phase starts: -1. If any requested mixin is not provided by the stack, the lifecycle will compare the missing mixins to the static list of mixins provided by stack buildpacks. If any mixins are still not provided, the build will fail. +1. If any requested mixin is not provided by the stack, the lifecycle will compare the missing mixins to the static list of mixins provided by stack buildpacks in accordance with [stage-specific mixin rules](https://github.com/buildpacks/rfcs/pull/109). If any mixins are still not provided, the build will fail. 1. The lifecycle will run the detect phase for all stackpacks defined in the builder. 1. The lifecycle will execute the stack buildpack build phase for all passing stackpack(s). If no stack buildpacks pass detect, the build will continue the build phase as normal (running userspace buildpacks). 1. After, during, or before the lifecycle's build phase, the lifecycle will begin a new phase called _extend_. From 7742fcb1693293ca148507da7239ad44b7170630 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Fri, 11 Sep 2020 16:24:14 -0500 Subject: [PATCH 050/140] Improve description of how mixins are resolved Signed-off-by: Joe Kutner --- text/0000-stack-buildpacks.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index 4efa762c2..38cfdf645 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -33,7 +33,8 @@ A stackpack may also define a list of mixins that it provides to the stack, or i A stack provider may choose to include stack buildpacks with the stack they distribute. If a stack includes any stack buildpacks, the following will occur when the build phase starts: -1. If any requested mixin is not provided by the stack, the lifecycle will compare the missing mixins to the static list of mixins provided by stack buildpacks in accordance with [stage-specific mixin rules](https://github.com/buildpacks/rfcs/pull/109). If any mixins are still not provided, the build will fail. +1. The platform may compare the list of requested mixins with the static list of mixins provided by the stack buildpacks, and fail the build if it choses to do so. +1. The lifecycle will compare the list of required mixins to the list of mixins provided by stack and stack buildpacks in accordance with [stage-specific mixin rules](https://github.com/buildpacks/rfcs/pull/109). If any mixins are still not provided, the build will fail. 1. The lifecycle will run the detect phase for all stackpacks defined in the builder. 1. The lifecycle will execute the stack buildpack build phase for all passing stackpack(s). If no stack buildpacks pass detect, the build will continue the build phase as normal (running userspace buildpacks). 1. After, during, or before the lifecycle's build phase, the lifecycle will begin a new phase called _extend_. From d9128babdac0eb9de5fabfe81feb33fbc7e48c57 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Fri, 11 Sep 2020 16:25:02 -0500 Subject: [PATCH 051/140] replace builder with build-image Signed-off-by: Joe Kutner --- text/0000-stack-buildpacks.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index 38cfdf645..0535b19c1 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -35,7 +35,7 @@ A stack provider may choose to include stack buildpacks with the stack they dist 1. The platform may compare the list of requested mixins with the static list of mixins provided by the stack buildpacks, and fail the build if it choses to do so. 1. The lifecycle will compare the list of required mixins to the list of mixins provided by stack and stack buildpacks in accordance with [stage-specific mixin rules](https://github.com/buildpacks/rfcs/pull/109). If any mixins are still not provided, the build will fail. -1. The lifecycle will run the detect phase for all stackpacks defined in the builder. +1. The lifecycle will run the detect phase for all stackpacks defined in the build-image. 1. The lifecycle will execute the stack buildpack build phase for all passing stackpack(s). If no stack buildpacks pass detect, the build will continue the build phase as normal (running userspace buildpacks). 1. After, during, or before the lifecycle's build phase, the lifecycle will begin a new phase called _extend_. 1. During the extend phase, stack buildpacks that passed detection will run against the run images accordingly (see details below). From 26944bdc01dcc3ec8a86a37e4937281deac243b1 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Fri, 11 Sep 2020 16:27:50 -0500 Subject: [PATCH 052/140] Added more specifics about what makes stackpacks different from userspace buildpacks Signed-off-by: Joe Kutner --- text/0000-stack-buildpacks.md | 1 + 1 file changed, 1 insertion(+) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index 0535b19c1..b27167215 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -49,6 +49,7 @@ A stack provider may choose to include stack buildpacks with the stack they dist * Is run as the `root` user * Configured with `privileged = true` in the `buildpack.toml` * Can only create one layer +* May include, in the created layer, modifications to any part of the filesystem, excluding ``, ``, and `` directories * Does not have access to the application source code * Is run before all regular buildpacks * Is run against both the build and run images From a3f5a44605492ab141e753e50863ad04b24b4619 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Fri, 11 Sep 2020 21:27:53 -0500 Subject: [PATCH 053/140] Move mixins under stacks in buildpack.toml Signed-off-by: Joe Kutner --- text/0000-stack-buildpacks.md | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index b27167215..92f416371 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -84,8 +84,10 @@ A stack buildpack MAY define a set of mixins it provides in either of two ways: A stack buildpack MAY define a set of stack specific mixins in the `buildpack.toml` with the following schema: ```toml -[[buildpack.provides.stacks]] +[[stacks]] id = "" + +[stacks.provides] any = mixins = [ "mixin name" ] ``` @@ -335,19 +337,24 @@ Where: privileged = non-idempotent = -[[buildpack.provides.stacks]] -id = "" +[[stacks]] +id = "" + +[stacks.requires] +mixins = [ "mixin name" ] + +[stacks.provides] any = -mixins = [ "" ] +mixins = [ "mixin name" ] ``` * `privileged` - when set to `true`, the lifecycle will run this buildpack as the `root` user. * `non-idempotent` - when set to `true`, indicates that the buildpack is not idempotent. The lifecycle will provide a clean filesystem from the stack image(s) before each run (i.e. no cache). -Under the `[buildpack.mixins]` table: +Under the `[stacks.provides]` table: * `any` - a boolean that, when true, indicates that the buildpack can provide all mixins -* `names` - a list of names that match mixins provided by this buildpack +* `mixins` - a list of names that match mixins provided by this buildpack ## Build Plan (TOML) From ae241d2eb2bb288b8585cb5bdc432c2d2e0c7240 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Fri, 11 Sep 2020 21:28:08 -0500 Subject: [PATCH 054/140] Add detect and builder interface to spec changes Signed-off-by: Joe Kutner --- text/0000-stack-buildpacks.md | 86 +++++++++++++++++++++++++++++++++++ 1 file changed, 86 insertions(+) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index 92f416371..e74fe94c9 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -383,3 +383,89 @@ mixin = [or.requires.metadata] # buildpack-specific data ``` + +## Lifecycle Interface + +### Usage + +#### `detector` +The platform MUST execute `detector` in the **build environment** + +Usage: +``` +/cnb/lifecycle/detector \ + [-app ] \ + [-buildpacks ] \ + [-group ] \ + [-log-level ] \ + [-order ] \ + [-plan ] \ + [-platform ] \ + [-stack-buildpacks ] \ + [-stack-group ] +``` + +##### Inputs +| Input | Environment Variable | Default Value | Description +|-----------------------|-----------------------------|-------------------------|---------------------- +| `` | `CNB_APP_DIR` | `/workspace` | Path to application directory +| `` | `CNB_BUILDPACKS_DIR` | `/cnb/buildpacks` | Path to buildpacks directory (see [Buildpacks Directory Layout](#buildpacks-directory-layout)) +| `` | `CNB_GROUP_PATH` | `./group.toml` | Path to output group definition +| `` | `CNB_LOG_LEVEL` | `info` | Log Level +| `` | `CNB_ORDER_PATH` | `./order.toml` | Path to order definition (see [`order.toml`](#ordertoml-toml)) +| `` | `CNB_PLAN_PATH` | `./plan.toml` | Path to output resolved build plan +| `` | `CNB_PLATFORM_DIR` | `/platform` | Path to platform directory +| `` | `CNB_STACK_BUILDPACKS_DIR` | `/cnb/stack/buildpacks` | Path to stack buildpacks directory (see [Buildpacks Directory Layout] +| `` | `CNB_STACK_GROUP_PATH` | `./stack-group.toml` | Path to output group definition(#buildpacks-directory-layout)) + +##### Outputs +| Output | Description +|--------------------|---------------------------------------------- +| [exit status] | (see Exit Code table below for values) +| `/dev/stdout` | Logs (info) +| `/dev/stderr` | Logs (warnings, errors) +| `` | Detected buildpack group (see [`group.toml`](#grouptoml-toml)) +| `` | Resolved Build Plan (see [`plan.toml`](#plantoml-toml)) +| `` | Detected stack buildpack group (see [`group.toml`](#grouptoml-toml)) + +#### `builder` +The platform MUST execute `builder` in the **build environment** + +Usage: +``` +/cnb/lifecycle/builder \ + [-app ] \ + [-buildpacks ] \ + [-group ] \ + [-layers ] \ + [-log-level ] \ + [-plan ] \ + [-platform ] \ + [-stack-buildpacks ] \ + [-stack-group ] +``` + +##### Inputs +| Input | Environment Variable | Default Value | Description +|-----------------------|-----------------------------|-------------------------|---------------------- +| `` | `CNB_APP_DIR` | `/workspace` | Path to application directory +| `` | `CNB_BUILDPACKS_DIR` | `/cnb/buildpacks` | Path to buildpacks directory (see [Buildpacks Directory Layout](#buildpacks-directory-layout)) +| `` | `CNB_GROUP_PATH` | `./group.toml` | Path to output group definition +| `` | `CNB_LAYERS_DIR` | `/layers` | Path to layers directory +| `` | `CNB_LOG_LEVEL` | `info` | Log Level +| `` | `CNB_ORDER_PATH` | `./order.toml` | Path to order definition (see [`order.toml`](#ordertoml-toml)) +| `` | `CNB_PLAN_PATH` | `./plan.toml` | Path to output resolved build plan +| `` | `CNB_PLATFORM_DIR` | `/platform` | Path to platform directory +| `` | `CNB_STACK_BUILDPACKS_DIR` | `/cnb/stack/buildpacks` | Path to stack buildpacks directory (see [Buildpacks Directory Layout] +| `` | `CNB_STACK_GROUP_PATH` | `./stack-group.toml` | Path to output group definition(#buildpacks-directory-layout)) + +##### Outputs +| Output | Description +|--------------------------------------------|---------------------------------------------- +| [exit status] | (see Exit Code table below for values) +| `/dev/stdout` | Logs (info) +| `/dev/stderr` | Logs (warnings, errors) +| `//` | Layer contents (see [Buildpack Interface Specfication](buildpack.md) +| `//.toml` | Layer metadata (see [Buildpack Interface Specfication](buildpack.md) +| `/.tgz` | Layer snapshot (see [Buildpack Interface Specfication](buildpack.md) +| `/config/metadata.toml` | Build metadata (see [`metadata.toml`](#metadatatoml-toml)) \ No newline at end of file From 941666719ec38934ee9394cc2599aa2e3567bb49 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Fri, 11 Sep 2020 21:28:08 -0500 Subject: [PATCH 055/140] Add detect and builder interface to spec changes Signed-off-by: Joe Kutner --- text/0000-stack-buildpacks.md | 65 +++++++++++++---------------------- 1 file changed, 23 insertions(+), 42 deletions(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index e74fe94c9..d164f5a75 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -394,13 +394,7 @@ The platform MUST execute `detector` in the **build environment** Usage: ``` /cnb/lifecycle/detector \ - [-app ] \ - [-buildpacks ] \ - [-group ] \ - [-log-level ] \ - [-order ] \ - [-plan ] \ - [-platform ] \ + ... [-stack-buildpacks ] \ [-stack-group ] ``` @@ -408,39 +402,24 @@ Usage: ##### Inputs | Input | Environment Variable | Default Value | Description |-----------------------|-----------------------------|-------------------------|---------------------- -| `` | `CNB_APP_DIR` | `/workspace` | Path to application directory -| `` | `CNB_BUILDPACKS_DIR` | `/cnb/buildpacks` | Path to buildpacks directory (see [Buildpacks Directory Layout](#buildpacks-directory-layout)) -| `` | `CNB_GROUP_PATH` | `./group.toml` | Path to output group definition -| `` | `CNB_LOG_LEVEL` | `info` | Log Level -| `` | `CNB_ORDER_PATH` | `./order.toml` | Path to order definition (see [`order.toml`](#ordertoml-toml)) -| `` | `CNB_PLAN_PATH` | `./plan.toml` | Path to output resolved build plan -| `` | `CNB_PLATFORM_DIR` | `/platform` | Path to platform directory +| ... | | | | `` | `CNB_STACK_BUILDPACKS_DIR` | `/cnb/stack/buildpacks` | Path to stack buildpacks directory (see [Buildpacks Directory Layout] -| `` | `CNB_STACK_GROUP_PATH` | `./stack-group.toml` | Path to output group definition(#buildpacks-directory-layout)) +| `` | `CNB_STACK_GROUP_PATH` | `./stack-group.toml` | Path to output group definition(#buildpacks-directory-layout)) ##### Outputs | Output | Description |--------------------|---------------------------------------------- -| [exit status] | (see Exit Code table below for values) -| `/dev/stdout` | Logs (info) -| `/dev/stderr` | Logs (warnings, errors) -| `` | Detected buildpack group (see [`group.toml`](#grouptoml-toml)) -| `` | Resolved Build Plan (see [`plan.toml`](#plantoml-toml)) | `` | Detected stack buildpack group (see [`group.toml`](#grouptoml-toml)) #### `builder` The platform MUST execute `builder` in the **build environment** +If the `` IS provided, the `builder` MUST be run with root privileges. If the the `` IS NOT provided, the `builder` MUST NOT be run with root privileges. + Usage: ``` /cnb/lifecycle/builder \ - [-app ] \ - [-buildpacks ] \ - [-group ] \ - [-layers ] \ - [-log-level ] \ - [-plan ] \ - [-platform ] \ + ... [-stack-buildpacks ] \ [-stack-group ] ``` @@ -448,24 +427,26 @@ Usage: ##### Inputs | Input | Environment Variable | Default Value | Description |-----------------------|-----------------------------|-------------------------|---------------------- -| `` | `CNB_APP_DIR` | `/workspace` | Path to application directory -| `` | `CNB_BUILDPACKS_DIR` | `/cnb/buildpacks` | Path to buildpacks directory (see [Buildpacks Directory Layout](#buildpacks-directory-layout)) -| `` | `CNB_GROUP_PATH` | `./group.toml` | Path to output group definition -| `` | `CNB_LAYERS_DIR` | `/layers` | Path to layers directory -| `` | `CNB_LOG_LEVEL` | `info` | Log Level -| `` | `CNB_ORDER_PATH` | `./order.toml` | Path to order definition (see [`order.toml`](#ordertoml-toml)) -| `` | `CNB_PLAN_PATH` | `./plan.toml` | Path to output resolved build plan -| `` | `CNB_PLATFORM_DIR` | `/platform` | Path to platform directory +| ... | | | | `` | `CNB_STACK_BUILDPACKS_DIR` | `/cnb/stack/buildpacks` | Path to stack buildpacks directory (see [Buildpacks Directory Layout] -| `` | `CNB_STACK_GROUP_PATH` | `./stack-group.toml` | Path to output group definition(#buildpacks-directory-layout)) +| `` | `CNB_STACK_GROUP_PATH` | `./stack-group.toml` | Path to output group definition(#buildpacks-directory-layout)) ##### Outputs | Output | Description |--------------------------------------------|---------------------------------------------- -| [exit status] | (see Exit Code table below for values) -| `/dev/stdout` | Logs (info) -| `/dev/stderr` | Logs (warnings, errors) -| `//` | Layer contents (see [Buildpack Interface Specfication](buildpack.md) -| `//.toml` | Layer metadata (see [Buildpack Interface Specfication](buildpack.md) +| ... | | `/.tgz` | Layer snapshot (see [Buildpack Interface Specfication](buildpack.md) -| `/config/metadata.toml` | Build metadata (see [`metadata.toml`](#metadatatoml-toml)) \ No newline at end of file + +#### `exporter` +Usage: +``` +/cnb/lifecycle/exporter \ + [-stack-group ] \ + ... +``` + +##### Inputs +| Input | Environment Variable | Default Value | Description +|---------------------|-----------------------|---------------------|--------------------------------------- +| ... | | | +| `` | `CNB_STACK_GROUP_PATH` | `./stack-group.toml` | Path to stack-group file (see [`group.toml`](#grouptoml-toml)) From 6ab64759f6158cfa807f7f49ecef8eddc4c15b6a Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Sat, 12 Sep 2020 07:45:23 -0500 Subject: [PATCH 056/140] Added a section on caching and restoring Signed-off-by: Joe Kutner --- text/0000-stack-buildpacks.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index d164f5a75..6a109726a 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -110,6 +110,14 @@ mixin = A userspace buildpack MAY NOT provide mixins in the build plan. +## Caching and Restoring + +During the export phase, the lifecycle will store any snapshot layers created during the build phase in the cache. + +During the restore phase of the next build, the lifecycle will download the snapshot layer with the cache and store it as a tarball in the `` directory (i.e. it will not extract it). The `restorer` cannot extract the snapshot because it will not run with root privileges. In addition, the `restorer` may run in a different container than the build, which means changes made to the base image are not guaranteed to carry forward. + +During the build phase, the lifecycle will extract and apply each snapshot tarball before running stack buildpacks, but after a snapshot-baseline has been captured. This ensures that all changes from the previous snapshot are preserved even if the stack buildpack does not make any additional changes. + ## Rebasing an App Before a launch image is rebased, the platform must re-run the any stackpacks that were used to build the launch image against the new run-image. Then, the rebase operation can be performed as normal, while including the stackpack layers as part of the stack. This will be made possible by including the stackpack in the run-image, but because the stackpack detect phase is not run, the operation does not need access to the application source. From 09316a1010c301a73bfc4dceae05fb9212e413c5 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Tue, 15 Sep 2020 08:41:59 -0500 Subject: [PATCH 057/140] Added details of how excludes and cached work Signed-off-by: Joe Kutner --- text/0000-stack-buildpacks.md | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index 6a109726a..412558bb2 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -72,7 +72,13 @@ A stack can provide stackpacks by including them in the `/cnb/stack/buildpacks` A stackpack will only execute if it passes detection. When the stackpack is executed, its detect and build scripts use the same parameters as the regular buildpacks. But the arguments may differ. For example, the first positional argument to the `bin/build` (the `` directory) MUST NOT be a standard layers directory adhering to the [Buildpacks Build specification](https://github.com/buildpacks/spec/blob/main/buildpack.md#build). -The stackpack's snapshot layer may be enriched by writing a `launch.toml` file. The `launch.toml` will define paths that will be restored even when the base image changes (ex. package indicies) and paths that will be excluded from the launch image (ex. `/var/cache`). +The stackpack's snapshot layer may be enriched by writing a `launch.toml` file. The `launch.toml` may define globs of files to be excluded from the image when it is _used_. Any excluded path may also be marked as _cached_, so that those excluded paths are recovered when the image is _used_. The term _used_ is defined as: + +* *Used for build-time build*: A given path is excluded at normal buildpack build-time, and recovered the next time the build image is extended with the stackpack. +* *Used for build-time run*: A given path is excluded from the final image, and recovered the next time the run image is extended with the stackpack (either rebase or rebuild). +* *Used for rebase run*: A given path is excluded from the rebased image, and recovered the next time the run image is extended with the stackpack (either rebase or rebuild). + +For example, a stack buildpack may choose to exclude `/var/cache` from the final run image, but may want to mark it as _cached_ to have it restored during build-time and rebase. ## Providing Mixins @@ -336,6 +342,10 @@ Where: * `paths` = a list of paths to exclude from the layer * `cache` = if true, the paths will be excluded from the launch image layer, but will be included in the cache layer. +1. Paths not referenced by an `[[excludes]]` entry will be included in the cache _and_ run image (default). +1. Any paths with an `[[excludes]]` entry and `cache = true` will be included in the cache image, but not the run image. +1. Any paths with an `[[excludes]]` entry and `cache = false` will not be included in the cache image or the run image. + ## buildpack.toml (TOML) This proposal adds new keys to the `[buildpack]` table in `buildpack.toml`, and a new `[[mixins]]` array of tables: From 672c06576a8b15c8d90a33267c6629420c7b517b Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Tue, 15 Sep 2020 13:58:52 -0500 Subject: [PATCH 058/140] Added rebaser spec changes Signed-off-by: Joe Kutner --- text/0000-stack-buildpacks.md | 19 ++++++++++++++++++- 1 file changed, 18 insertions(+), 1 deletion(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index 412558bb2..c61ead30a 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -126,7 +126,9 @@ During the build phase, the lifecycle will extract and apply each snapshot tarba ## Rebasing an App -Before a launch image is rebased, the platform must re-run the any stackpacks that were used to build the launch image against the new run-image. Then, the rebase operation can be performed as normal, while including the stackpack layers as part of the stack. This will be made possible by including the stackpack in the run-image, but because the stackpack detect phase is not run, the operation does not need access to the application source. +Before a launch image is rebased, the platform must re-run the any stackpacks that were used to build the launch image against the new run-image. The image containing the stack buildpacks and the builder binary must be provided to the rebaser operation as an argument. A platform may choose to provide the same stack buildpacks and builder binary used during the build that create the launch-image being rebased, or it may provide updates versions (which may increase the risk of something failing in the rebase process). + +Then, the rebase operation can be performed as normal, while including the stackpack layers as part of the stack. This will be made possible by including the stackpack in the run-image, but because the stackpack detect phase is not run, the operation does not need access to the application source. ## Example: Apt Buildpack @@ -468,3 +470,18 @@ Usage: |---------------------|-----------------------|---------------------|--------------------------------------- | ... | | | | `` | `CNB_STACK_GROUP_PATH` | `./stack-group.toml` | Path to stack-group file (see [`group.toml`](#grouptoml-toml)) + + +#### `rebaser` +Usage: +``` +/cnb/lifecycle/exporter \ + [-build-image ] \ + ... +``` + +##### Inputs +| Input | Environment Variable | Default Value | Description +|---------------------|-----------------------|------------------------|--------------------------------------- +| ... | | | +| `` | `CNB_BUILD_IMAGE` | derived from `` | Image containing `builder` and stack buildpacks From f6239c55d545bc790256dc4c37d0e6d8da04faee Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Wed, 16 Sep 2020 10:16:39 -0500 Subject: [PATCH 059/140] Added extender spec changes Signed-off-by: Joe Kutner --- text/0000-stack-buildpacks.md | 58 ++++++++++++++++++++++++++++++++++- 1 file changed, 57 insertions(+), 1 deletion(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index c61ead30a..7985a6266 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -128,8 +128,19 @@ During the build phase, the lifecycle will extract and apply each snapshot tarba Before a launch image is rebased, the platform must re-run the any stackpacks that were used to build the launch image against the new run-image. The image containing the stack buildpacks and the builder binary must be provided to the rebaser operation as an argument. A platform may choose to provide the same stack buildpacks and builder binary used during the build that create the launch-image being rebased, or it may provide updates versions (which may increase the risk of something failing in the rebase process). +A platform may choose to store the stack buildpacks and builder binary in any of the following images: +* A companion image of the launch-image +* A builder image that is available when rebasing +* The launch-image itself (store the stack buildpacks and builder binary alongside the application itself) + Then, the rebase operation can be performed as normal, while including the stackpack layers as part of the stack. This will be made possible by including the stackpack in the run-image, but because the stackpack detect phase is not run, the operation does not need access to the application source. +## Extending the Run Image + +We will introduce a new phase to the lifecycle to support extending the run-image with stack buildpacks. The _extend_ phase will be responsible for running stack buildpacks on the run-image, and creating layers that will be applied to the launch-image. + + + ## Example: Apt Buildpack (**note**: this is only an example. A real Apt Buildpack would probably be more robust). @@ -475,7 +486,7 @@ Usage: #### `rebaser` Usage: ``` -/cnb/lifecycle/exporter \ +/cnb/lifecycle/rebaser \ [-build-image ] \ ... ``` @@ -485,3 +496,48 @@ Usage: |---------------------|-----------------------|------------------------|--------------------------------------- | ... | | | | `` | `CNB_BUILD_IMAGE` | derived from `` | Image containing `builder` and stack buildpacks + + +#### `extender` +Usage: +``` +/cnb/lifecycle/extender \ + [-stack-buildpacks ] \ + [-stack-group ] \ + [-layers ] \ + [-log-level ] \ + [-plan ] \ + [-platform ] +``` + +##### Inputs +| Input | Env | Default Value | Description +|----------------|-----------------------|-------------------|---------------------- +| `` | `CNB_LAYERS_DIR` | `/layers` | Path to layers directory +| `` | `CNB_LOG_LEVEL` | `info` | Log Level +| `` | `CNB_PLAN_PATH` | `./plan.toml` | Path to resolved build plan (see [`plan.toml`](#plantoml-toml)) +| `` | `CNB_PLATFORM_DIR` | `/platform` | Path to platform directory +| `` | `CNB_STACK_BUILDPACKS_DIR` | `/cnb/stack/buildpacks` | Path to stack buildpacks directory (see [Buildpacks Directory Layout] +| `` | `CNB_STACK_GROUP_PATH` | `./stack-group.toml` | Path to output group definition(#buildpacks-directory-layout)) + +##### Outputs +| Output | Description +|--------------------------------------------|---------------------------------------------- +| [exit status] | (see Exit Code table below for values) +| `/dev/stdout` | Logs (info) +| `/dev/stderr` | Logs (warnings, errors) +| `/.tgz` | Layer snapshot (see [Buildpack Interface Specfication](buildpack.md) +| `/config/metadata.toml` | Build metadata (see [`metadata.toml`](#metadatatoml-toml)) + +| Exit Code | Result| +|-----------|-------| +| `0` | Success +| `11` | Platform API incompatibility error +| `12` | Buildpack API incompatibility error +| `1-10`, `13-99` | Generic lifecycle errors +| `401` | Buildpack build error +| `400`, `402-499`| Build-specific lifecycle errors + +- The lifecycle SHALL execute all stack buildpacks in the order defined in `` according to the process outlined in the [Buildpack Interface Specification](buildpack.md). +- The lifecycle SHALL add all invoked stack buildpacks to`/config/metadata.toml`. +- The lifecycle SHALL aggregate all `processes` and BOM entries returned by buildpacks in `/config/metadata.toml`. \ No newline at end of file From 66d5f08cdd25ffd872ba85692dfc79883872fba5 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Wed, 16 Sep 2020 10:23:49 -0500 Subject: [PATCH 060/140] Fix toml formating of schema def Signed-off-by: Joe Kutner --- text/0000-stack-buildpacks.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index 7985a6266..92223a708 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -89,7 +89,7 @@ A stack buildpack MAY define a set of mixins it provides in either of two ways: A stack buildpack MAY define a set of stack specific mixins in the `buildpack.toml` with the following schema: -```toml +``` [[stacks]] id = "" From a182de3de5e5515952a31b670f7ca1f1f5ff3427 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Wed, 16 Sep 2020 10:32:00 -0500 Subject: [PATCH 061/140] Updated unresolved questions and added future work Signed-off-by: Joe Kutner --- text/0000-stack-buildpacks.md | 24 +++++++++++++++++------- 1 file changed, 17 insertions(+), 7 deletions(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index 92223a708..f1fef049d 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -56,7 +56,13 @@ A stack provider may choose to include stack buildpacks with the stack they dist * Is distributed with the stack run and/or build image * May not create layers using the `` directory -The stackpack interface is identical to the buildpack interface (i.e. the same `bin/detect` and `bin/build` scripts are required). However, some of the context it is run in is different from regular buildpacks. +The stackpack interface is similar to the buildpack interface +* the same `bin/detect` and `bin/build` scripts are required +* the `bin/detect` will have read-only access to the app +* The positional arguments for `bin/detect` and `bin/build` are the same (though the values may be different) +* The environment variables and inputs for `bin/detect` and `bin/build` are the same (though the values may be different) + +However, some of the context it is run in is different from regular buildpacks. For each stackpack, the lifecycle will use [snapshotting](https://github.com/GoogleContainerTools/kaniko/blob/master/docs/designdoc.md#snapshotting-snapshotting) to capture changes made during the stackpack's build phase excluding the a few specific directories and files. @@ -291,6 +297,12 @@ mixin = true name = "jq" ``` +## Future work + +In the future, we plan to enhance the stack buildpack interface with the following: + +* A `CNB_STACK_TYPE` env var that a stack buildpack can use to behave differently on each part of the stack + # Drawbacks [drawbacks]: #drawbacks @@ -311,12 +323,10 @@ name = "jq" # Unresolved Questions [unresolved-questions]: #unresolved-questions -- Do we want (or need) a `CNB_STACK_TYPE` env var so that a stack buildpack can behave differently on each part of the stack? -- Should the stackpack's detect have read-only access to the app? - - This would likely be driven by a stackpack that does not provide mixins, but instead dynamically contributes to the build plan based on the contents of the app source code. I don't know if we have a use case for this, but I can imaging a buildpack that reads environment variables as input to some function. -- Should stackpacks be able to define per-stack mixins? - - We could support a top-level `mixins` array, and allow refinements under `[[stacks]] mixins`. If we do this, we need to distinguish between provided and required mixins (at least under `[[stacks]]`). - - If userspace buildpacks can require mixins from `bin/detect`, the stackpack could use this mechanism to require per-stack mixins. +- what happens if a mixin that already exist on the image is required, how does the lifecycle know? +- do we need/want distinct upgrade vs rebase operations (where one re-runs stackpacks, and one does not)? +- what about the bill of materials? +- what metadata do we need to do an upgrade, which what are the schema of the labels? # Spec. Changes (OPTIONAL) [spec-changes]: #spec-changes From 876af605af959a0c619ae05388dc5b22f4fe5d44 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Thu, 17 Sep 2020 09:42:20 -0500 Subject: [PATCH 062/140] Added note about future work for creator Signed-off-by: Joe Kutner --- text/0000-stack-buildpacks.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index f1fef049d..3358527a1 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -145,8 +145,6 @@ Then, the rebase operation can be performed as normal, while including the stack We will introduce a new phase to the lifecycle to support extending the run-image with stack buildpacks. The _extend_ phase will be responsible for running stack buildpacks on the run-image, and creating layers that will be applied to the launch-image. - - ## Example: Apt Buildpack (**note**: this is only an example. A real Apt Buildpack would probably be more robust). @@ -302,6 +300,7 @@ name = "jq" In the future, we plan to enhance the stack buildpack interface with the following: * A `CNB_STACK_TYPE` env var that a stack buildpack can use to behave differently on each part of the stack +* Support for `creator`. Because of the new extend phase, it is not possible to easily run the entire buildpack process with a single binary. # Drawbacks [drawbacks]: #drawbacks From 08fcc16d0c035e64075a75d3acf1e33712a79e97 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Thu, 17 Sep 2020 09:54:30 -0500 Subject: [PATCH 063/140] Added section describing Resolving Mixins from stackpacks Signed-off-by: Joe Kutner --- text/0000-stack-buildpacks.md | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index 3358527a1..6dd0cdebd 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -86,7 +86,9 @@ The stackpack's snapshot layer may be enriched by writing a `launch.toml` file. For example, a stack buildpack may choose to exclude `/var/cache` from the final run image, but may want to mark it as _cached_ to have it restored during build-time and rebase. -## Providing Mixins +## Mixins + +### Providing Mixins A stack buildpack MAY define a set of mixins it provides in either of two ways: @@ -104,12 +106,13 @@ any = mixins = [ "mixin name" ] ``` -Additionally, mixins MAY be dyncamically provided in the build plan: +Additionally, mixins MAY be dynamically provided in the build plan: ``` [[provides]] name = "" -mixin = +any = +mixin = ``` A userspace buildpack MAY require mixins in the build plan @@ -122,6 +125,15 @@ mixin = A userspace buildpack MAY NOT provide mixins in the build plan. +### Resolving Mixins + +After the detect phase, the lifecycle will merge the list of provided mixins from the following sources: +* `stack.toml` +* `buildpack.toml` of any stack buildpacks +* The build plan (any `[[provides]]` tables with `mixin = true`) + +If any required mixins from the Build Plan (any `[[required]]` tables with `mixin = true`) are not provided, then the build will fail. Otherwise the build will continue. + ## Caching and Restoring During the export phase, the lifecycle will store any snapshot layers created during the build phase in the cache. From cae373126f861523d66439663329b2da1524987e Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Thu, 17 Sep 2020 09:55:47 -0500 Subject: [PATCH 064/140] Update open questions for stack packs Signed-off-by: Joe Kutner --- text/0000-stack-buildpacks.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index 6dd0cdebd..f7c5d0020 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -335,9 +335,9 @@ In the future, we plan to enhance the stack buildpack interface with the followi [unresolved-questions]: #unresolved-questions - what happens if a mixin that already exist on the image is required, how does the lifecycle know? -- do we need/want distinct upgrade vs rebase operations (where one re-runs stackpacks, and one does not)? - what about the bill of materials? -- what metadata do we need to do an upgrade, which what are the schema of the labels? +- what metadata do we need to do an rebase; what are the schema of the labels? +- how do we prevent problems on rebase if the mixins provided by the stack change? # Spec. Changes (OPTIONAL) [spec-changes]: #spec-changes From e9d9100546bbf3660c88260c8c53e054bd1d9333 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Thu, 17 Sep 2020 10:01:02 -0500 Subject: [PATCH 065/140] Added an open question for stackpacks Signed-off-by: Joe Kutner --- text/0000-stack-buildpacks.md | 1 + 1 file changed, 1 insertion(+) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index f7c5d0020..9e36b23a7 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -338,6 +338,7 @@ In the future, we plan to enhance the stack buildpack interface with the followi - what about the bill of materials? - what metadata do we need to do an rebase; what are the schema of the labels? - how do we prevent problems on rebase if the mixins provided by the stack change? +- While the rebase process may get it's stackpacks from any image passed to it, do we need to support official ways of producing well-formed run-builders? # Spec. Changes (OPTIONAL) [spec-changes]: #spec-changes From aee24d3b0871a9ce7fc51de273ee91a66257117d Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Thu, 17 Sep 2020 10:01:30 -0500 Subject: [PATCH 066/140] Open Qs cleanup Signed-off-by: Joe Kutner --- text/0000-stack-buildpacks.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index 9e36b23a7..23b3ffb96 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -336,9 +336,9 @@ In the future, we plan to enhance the stack buildpack interface with the followi - what happens if a mixin that already exist on the image is required, how does the lifecycle know? - what about the bill of materials? -- what metadata do we need to do an rebase; what are the schema of the labels? +- what metadata do we need to do a rebase; what are the schema of the labels? - how do we prevent problems on rebase if the mixins provided by the stack change? -- While the rebase process may get it's stackpacks from any image passed to it, do we need to support official ways of producing well-formed run-builders? +- while the rebase process may get its stackpacks from any image passed to it, do we need to support official ways of producing well-formed run-builders? # Spec. Changes (OPTIONAL) [spec-changes]: #spec-changes From 253ce6395927ee5dc0f349f9e6d5214f708ce23e Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Fri, 18 Sep 2020 15:45:04 -0500 Subject: [PATCH 067/140] Update apt buildpack example for mixins Signed-off-by: Joe Kutner --- text/0000-stack-buildpacks.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index 23b3ffb96..a0213de3a 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -168,11 +168,11 @@ A buildpack that can install an arbitrary list of mixins would have a `buildpack id = "example/apt" privileged = true -[buildpack.mixins] -any = true - [[stacks]] id = "io.buildpacks.stacks.bionic" + +[stacks.provides] +any = true ``` Its `bin/detect` would have the following contents: From 0f31a43e09bd37f0370bd36c467b9c30aef89d8b Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Mon, 21 Sep 2020 16:17:17 -0500 Subject: [PATCH 068/140] Add more detail of mixins in the build plan Signed-off-by: Joe Kutner --- text/0000-stack-buildpacks.md | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index a0213de3a..a4b62da0a 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -125,6 +125,10 @@ mixin = A userspace buildpack MAY NOT provide mixins in the build plan. +### Requiring Mixins + +Stack buildpacks MUST NOT require any entries in the build plan (neither mixins nor non-mixins). This ensures that we do not need to re-run the detect phase. + ### Resolving Mixins After the detect phase, the lifecycle will merge the list of provided mixins from the following sources: @@ -134,6 +138,20 @@ After the detect phase, the lifecycle will merge the list of provided mixins fro If any required mixins from the Build Plan (any `[[required]]` tables with `mixin = true`) are not provided, then the build will fail. Otherwise the build will continue. +During the build phase, the lifecycle will create a build plan containing only the entries required during that stage (build or run). +* If a mixin is required for "run" stage only, it will not appear in the build plan entries during build +* If a mixin is required for "build" stage only, it will not appear in the build plan entries during extend + +The entries of the build plan during the build and extend phases will have a new `mixin` key. For example: + +``` +[[entries]] +name = "libpq" +mixin = true +``` + +A Stack Buildpack that needs to install mixins must select them from the build plan. + ## Caching and Restoring During the export phase, the lifecycle will store any snapshot layers created during the build phase in the cache. @@ -437,6 +455,17 @@ mixin = # buildpack-specific data ``` +### Buildpack Plan (TOML) + +```toml +[[entries]] +name = "" +mixin = + +[entries.metadata] +# buildpack-specific data +``` + ## Lifecycle Interface ### Usage From f49156186ad00c659ccfe9abd8bbeebd8159d4c0 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Wed, 23 Sep 2020 08:06:42 -0500 Subject: [PATCH 069/140] More detail on stackpack mixin resolution Signed-off-by: Joe Kutner --- text/0000-stack-buildpacks.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index a4b62da0a..cf10df3b4 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -138,6 +138,8 @@ After the detect phase, the lifecycle will merge the list of provided mixins fro If any required mixins from the Build Plan (any `[[required]]` tables with `mixin = true`) are not provided, then the build will fail. Otherwise the build will continue. +If a stack buildpack provides a mixin that is not required, the stack buildpack MAY pass detection. This is in contrast to a userspace buildpack providing a dependency that is not required, which fails detection. If a mixin is required for a [single stage only](https://github.com/buildpacks/rfcs/pull/109) with the `build:` or `run:` prefix, a stack buildpack may provide it for both stages without failing detection. However, it will not be included in the Buildpack Build Plan during the stage where it is not required. + During the build phase, the lifecycle will create a build plan containing only the entries required during that stage (build or run). * If a mixin is required for "run" stage only, it will not appear in the build plan entries during build * If a mixin is required for "build" stage only, it will not appear in the build plan entries during extend From 87e455e1bc4dfbf17f0385a4c0a526b353b4de2d Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Thu, 1 Oct 2020 13:13:19 -0500 Subject: [PATCH 070/140] Add order.toml for stackpacks Signed-off-by: Joe Kutner --- text/0000-stack-buildpacks.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index cf10df3b4..f942f11d6 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -68,7 +68,7 @@ For each stackpack, the lifecycle will use [snapshotting](https://github.com/Goo Additionally, a platform may store a new stack image to cache the changes. All of the captured changes will be included in a single layer produced as output from the stackpack. The `/layers` dir MAY NOT be used to create arbitrary layers. -A stack can provide stackpacks by including them in the `/cnb/stack/buildpacks` directory. By default, an implicit order will be created for all stackpacks included in a stack. The order can be overridden in the `builder.toml` with the following configuration: +A stack can provide stackpacks by including them in the `/cnb/stack/buildpacks` directory, and providing an `/cnb/stack/buildpacks/order.toml` (following the [`order.toml` schema](https://github.com/buildpacks/spec/blob/main/platform.md#ordertoml-toml)) to define their order of execution. The order can be overridden in the `builder.toml` with the following configuration: ``` [[stack.order]] From 7baf37f6b5d2c1600bf0419f397abe6862fe9d1c Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Thu, 1 Oct 2020 13:15:31 -0500 Subject: [PATCH 071/140] Update text/0000-stack-buildpacks.md Signed-off-by: Joe Kutner Co-authored-by: Yael Harel <43007598+yaelharel@users.noreply.github.com> --- text/0000-stack-buildpacks.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index f942f11d6..6d7b2ad3b 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -58,7 +58,7 @@ A stack provider may choose to include stack buildpacks with the stack they dist The stackpack interface is similar to the buildpack interface * the same `bin/detect` and `bin/build` scripts are required -* the `bin/detect` will have read-only access to the app +* The `bin/detect` will have read-only access to the app * The positional arguments for `bin/detect` and `bin/build` are the same (though the values may be different) * The environment variables and inputs for `bin/detect` and `bin/build` are the same (though the values may be different) @@ -593,4 +593,4 @@ Usage: - The lifecycle SHALL execute all stack buildpacks in the order defined in `` according to the process outlined in the [Buildpack Interface Specification](buildpack.md). - The lifecycle SHALL add all invoked stack buildpacks to`/config/metadata.toml`. -- The lifecycle SHALL aggregate all `processes` and BOM entries returned by buildpacks in `/config/metadata.toml`. \ No newline at end of file +- The lifecycle SHALL aggregate all `processes` and BOM entries returned by buildpacks in `/config/metadata.toml`. From f5f022e30fdadcad0896749fad64c43a03842a1f Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Thu, 1 Oct 2020 13:15:45 -0500 Subject: [PATCH 072/140] Update text/0000-stack-buildpacks.md Signed-off-by: Joe Kutner Co-authored-by: Yael Harel <43007598+yaelharel@users.noreply.github.com> --- text/0000-stack-buildpacks.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index 6d7b2ad3b..1753b929d 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -33,7 +33,7 @@ A stackpack may also define a list of mixins that it provides to the stack, or i A stack provider may choose to include stack buildpacks with the stack they distribute. If a stack includes any stack buildpacks, the following will occur when the build phase starts: -1. The platform may compare the list of requested mixins with the static list of mixins provided by the stack buildpacks, and fail the build if it choses to do so. +1. The platform may compare the list of requested mixins with the static list of mixins provided by the stack buildpacks, and fail the build if it chooses to do so. 1. The lifecycle will compare the list of required mixins to the list of mixins provided by stack and stack buildpacks in accordance with [stage-specific mixin rules](https://github.com/buildpacks/rfcs/pull/109). If any mixins are still not provided, the build will fail. 1. The lifecycle will run the detect phase for all stackpacks defined in the build-image. 1. The lifecycle will execute the stack buildpack build phase for all passing stackpack(s). If no stack buildpacks pass detect, the build will continue the build phase as normal (running userspace buildpacks). From 58cecb8f9fe4a971c9f0e18f433722d973fbdf4d Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Thu, 1 Oct 2020 13:16:03 -0500 Subject: [PATCH 073/140] Update text/0000-stack-buildpacks.md Signed-off-by: Joe Kutner Co-authored-by: Yael Harel <43007598+yaelharel@users.noreply.github.com> --- text/0000-stack-buildpacks.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index 1753b929d..45ae128d7 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -57,7 +57,7 @@ A stack provider may choose to include stack buildpacks with the stack they dist * May not create layers using the `` directory The stackpack interface is similar to the buildpack interface -* the same `bin/detect` and `bin/build` scripts are required +* The same `bin/detect` and `bin/build` scripts are required * The `bin/detect` will have read-only access to the app * The positional arguments for `bin/detect` and `bin/build` are the same (though the values may be different) * The environment variables and inputs for `bin/detect` and `bin/build` are the same (though the values may be different) From 6e7f0cf8ef625f7ed5f34a717126e893b9d7459b Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Fri, 2 Oct 2020 09:40:09 -0500 Subject: [PATCH 074/140] Move def of stackpack in to what it is Signed-off-by: Joe Kutner --- text/0000-stack-buildpacks.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index 45ae128d7..291b7ee26 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -24,7 +24,7 @@ However, many applications and buildpacks require modifications to the stack the [what-it-is]: #what-it-is - *root buildpack* - a new type of buildpack that runs as the root user -- *stack buildpack* - a type of root buildpack that runs against the stack image(s) instead of an app. Stack buildpacks must not make changes to the build and run images that either violate stack [compatibility guarantees](https://github.com/buildpacks/spec/blob/main/platform.md#compatibility-guarantees) or violate the contract defined by that stack's author. +- *stack buildpack* - (a.k.a. stackpack) a type of root buildpack that runs against the stack image(s) instead of an app. Stack buildpacks must not make changes to the build and run images that either violate stack [compatibility guarantees](https://github.com/buildpacks/spec/blob/main/platform.md#compatibility-guarantees) or violate the contract defined by that stack's author. - *userspace buildpack* - the traditional definition of a buildpack (i.e. does not run as root, and runs against an app) A new type of buildpack, called a Stack buildpack, may run against a stack (both build and run images) in order to extend it in ways that are only possible by running privileged commands. Unlike userspace buildpacks, stack buildpack can modify any path on the filesystem. Userspace buildpack can only create/modify disjoint layers (either by adding a dir to `` or modifying an app slice), which makes possible features like individual layer reuse that is independent or ordering. @@ -44,7 +44,7 @@ A stack provider may choose to include stack buildpacks with the stack they dist # How it Works [how-it-works]: #how-it-works - A stack buildpack (a.k.a. stackpack) is a special case of buildpack that has the following properties: + A stack buildpack is a special case of buildpack that has the following properties: * Is run as the `root` user * Configured with `privileged = true` in the `buildpack.toml` From 49971b603767afdda32cd8b40ba2081ebb2033b5 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Fri, 2 Oct 2020 09:41:51 -0500 Subject: [PATCH 075/140] fix typo Signed-off-by: Joe Kutner --- text/0000-stack-buildpacks.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index 291b7ee26..a0e354374 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -11,7 +11,7 @@ # Summary [summary]: #summary -This is proposal for a new type of buildpack that runs against a stack in order to extend it in ways that are only possible by running privileged commands. +This is a proposal for a new type of buildpack that runs against a stack in order to extend it in ways that are only possible by running privileged commands. # Motivation [motivation]: #motivation From ff1c27174725118dfbd0c4ffa5baeb307155d9a2 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Fri, 2 Oct 2020 09:42:57 -0500 Subject: [PATCH 076/140] Fix formatting Signed-off-by: Joe Kutner --- text/0000-stack-buildpacks.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index a0e354374..c898c12ad 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -459,7 +459,7 @@ mixin = ### Buildpack Plan (TOML) -```toml +``` [[entries]] name = "" mixin = From 7a5365614ca4182dc75a1c6565b1137e30ab781b Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Mon, 12 Oct 2020 10:16:59 -0500 Subject: [PATCH 077/140] Update text/0000-stack-buildpacks.md Signed-off-by: Joe Kutner Co-authored-by: Yael Harel <43007598+yaelharel@users.noreply.github.com> --- text/0000-stack-buildpacks.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index c898c12ad..942514fbd 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -64,7 +64,7 @@ The stackpack interface is similar to the buildpack interface However, some of the context it is run in is different from regular buildpacks. -For each stackpack, the lifecycle will use [snapshotting](https://github.com/GoogleContainerTools/kaniko/blob/master/docs/designdoc.md#snapshotting-snapshotting) to capture changes made during the stackpack's build phase excluding the a few specific directories and files. +For each stackpack, the lifecycle will use [snapshotting](https://github.com/GoogleContainerTools/kaniko/blob/master/docs/designdoc.md#snapshotting-snapshotting) to capture changes made during the stackpack's build phase excluding a few specific directories and files. Additionally, a platform may store a new stack image to cache the changes. All of the captured changes will be included in a single layer produced as output from the stackpack. The `/layers` dir MAY NOT be used to create arbitrary layers. From ca9b034f9ff262222bd28a8773a0abdc4b5e2e71 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Mon, 12 Oct 2020 10:24:46 -0500 Subject: [PATCH 078/140] Added definition of launch image Signed-off-by: Joe Kutner --- text/0000-stack-buildpacks.md | 1 + 1 file changed, 1 insertion(+) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index 942514fbd..031260c48 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -26,6 +26,7 @@ However, many applications and buildpacks require modifications to the stack the - *root buildpack* - a new type of buildpack that runs as the root user - *stack buildpack* - (a.k.a. stackpack) a type of root buildpack that runs against the stack image(s) instead of an app. Stack buildpacks must not make changes to the build and run images that either violate stack [compatibility guarantees](https://github.com/buildpacks/spec/blob/main/platform.md#compatibility-guarantees) or violate the contract defined by that stack's author. - *userspace buildpack* - the traditional definition of a buildpack (i.e. does not run as root, and runs against an app) +- *launch image* - the final image produced by a buildpack build. It consists of a run-image combined with layers created by buildpacks. A new type of buildpack, called a Stack buildpack, may run against a stack (both build and run images) in order to extend it in ways that are only possible by running privileged commands. Unlike userspace buildpacks, stack buildpack can modify any path on the filesystem. Userspace buildpack can only create/modify disjoint layers (either by adding a dir to `` or modifying an app slice), which makes possible features like individual layer reuse that is independent or ordering. From c2c946f353804bba9f25a5ab0124c61cd2f1a4ac Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Mon, 12 Oct 2020 10:34:29 -0500 Subject: [PATCH 079/140] Added details about builder privileges Signed-off-by: Joe Kutner --- text/0000-stack-buildpacks.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index 031260c48..3a6687f51 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -159,7 +159,7 @@ A Stack Buildpack that needs to install mixins must select them from the build p During the export phase, the lifecycle will store any snapshot layers created during the build phase in the cache. -During the restore phase of the next build, the lifecycle will download the snapshot layer with the cache and store it as a tarball in the `` directory (i.e. it will not extract it). The `restorer` cannot extract the snapshot because it will not run with root privileges. In addition, the `restorer` may run in a different container than the build, which means changes made to the base image are not guaranteed to carry forward. +During the restore phase of the next build, the lifecycle will download the snapshot layer with the cache and store it as a tarball in the `` directory (i.e. it will not extract it). The `restorer` cannot extract the snapshot because it will not run with root privileges. In addition, the `restorer` may run in a different container than the build, which means changes made to the base image are not guaranteed to carry forward. The `builder` will run with root privileges, but will drop privileges after executing stackpacks and before running userspace buildpacks. During the build phase, the lifecycle will extract and apply each snapshot tarball before running stack buildpacks, but after a snapshot-baseline has been captured. This ensures that all changes from the previous snapshot are preserved even if the stack buildpack does not make any additional changes. From 83f2b1c35a0571ce499c13a0e3cc2ee387a1543c Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Wed, 14 Oct 2020 08:26:05 -0500 Subject: [PATCH 080/140] Update text/0000-stack-buildpacks.md Signed-off-by: Joe Kutner Co-authored-by: Yael Harel <43007598+yaelharel@users.noreply.github.com> --- text/0000-stack-buildpacks.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index 3a6687f51..557e6f32f 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -18,7 +18,7 @@ This is a proposal for a new type of buildpack that runs against a stack in orde Normally, buildpacks do not run as root. This is an intentional design decision that ensures operations like `rebase` will work on day-2. -However, many applications and buildpacks require modifications to the stack they run on, such as adding system packages or custom certificates. For this reason, we need a mechanism that buildpack authors and buildpack users and leverage to extend their stacks. +However, many applications and buildpacks require modifications to the stack they run on, such as adding system packages or custom certificates. For this reason, we need a mechanism that buildpack authors and buildpack users can leverage to extend their stacks. # What it is [what-it-is]: #what-it-is From 82eeb736d97182a5c0508de19e439b66d506c667 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Wed, 14 Oct 2020 08:39:41 -0500 Subject: [PATCH 081/140] Added project.toml to apt cnb example Signed-off-by: Joe Kutner --- text/0000-stack-buildpacks.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index 557e6f32f..5305d3331 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -222,6 +222,15 @@ cache = true EOF ``` +Based on the proposal in [App Mixins](https://github.com/buildpacks/rfcs/pull/110), the user's `project.toml` might include: + +```toml +[build] +mixins = [ "libpq" ] +``` + +This would instruct the Apt Buildpack to install the libpq package. + ## Example: CA Certificate Buildpack Support for custom CA Certificates can be accomplished with two buildpacks: a stackpack that can install the cert, and a normal buildpack that can provide a cert in the build plan. From 150ca4574f50b40640f06750cdcc4f62784ebaac Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Wed, 14 Oct 2020 08:40:31 -0500 Subject: [PATCH 082/140] Fix link Signed-off-by: Joe Kutner --- text/0000-stack-buildpacks.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index 5305d3331..ca929b8aa 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -222,7 +222,7 @@ cache = true EOF ``` -Based on the proposal in [App Mixins](https://github.com/buildpacks/rfcs/pull/110), the user's `project.toml` might include: +Based on the proposal in [App Mixins](https://github.com/buildpacks/rfcs/pull/112), the user's `project.toml` might include: ```toml [build] From c3bdf7782cc211c56b1ecbbfe1ee03a0c10849b3 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Wed, 14 Oct 2020 08:44:29 -0500 Subject: [PATCH 083/140] Update text/0000-stack-buildpacks.md Signed-off-by: Joe Kutner Co-authored-by: Natalie Arellano --- text/0000-stack-buildpacks.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index ca929b8aa..f78e8c922 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -51,7 +51,7 @@ A stack provider may choose to include stack buildpacks with the stack they dist * Configured with `privileged = true` in the `buildpack.toml` * Can only create one layer * May include, in the created layer, modifications to any part of the filesystem, excluding ``, ``, and `` directories -* Does not have access to the application source code +* Does not have access to the application source code during the `build` phase * Is run before all regular buildpacks * Is run against both the build and run images * Is distributed with the stack run and/or build image From 2148aea2951ee88bd92c90b1355b50fc649d5c26 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Wed, 14 Oct 2020 08:45:33 -0500 Subject: [PATCH 084/140] Fix location of order.toml Signed-off-by: Joe Kutner --- text/0000-stack-buildpacks.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index f78e8c922..c3adc6199 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -69,7 +69,7 @@ For each stackpack, the lifecycle will use [snapshotting](https://github.com/Goo Additionally, a platform may store a new stack image to cache the changes. All of the captured changes will be included in a single layer produced as output from the stackpack. The `/layers` dir MAY NOT be used to create arbitrary layers. -A stack can provide stackpacks by including them in the `/cnb/stack/buildpacks` directory, and providing an `/cnb/stack/buildpacks/order.toml` (following the [`order.toml` schema](https://github.com/buildpacks/spec/blob/main/platform.md#ordertoml-toml)) to define their order of execution. The order can be overridden in the `builder.toml` with the following configuration: +A stack can provide stackpacks by including them in the `/cnb/stack/buildpacks` directory, and providing an `/cnb/stack/order.toml` (following the [`order.toml` schema](https://github.com/buildpacks/spec/blob/main/platform.md#ordertoml-toml)) to define their order of execution. The order can be overridden in the `builder.toml` with the following configuration: ``` [[stack.order]] From 568ec229047e50aebcbc9b3608730388e9c846e7 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Sat, 17 Oct 2020 00:45:27 -0500 Subject: [PATCH 085/140] Update text/0000-stack-buildpacks.md Signed-off-by: Joe Kutner Co-authored-by: Emily Casey --- text/0000-stack-buildpacks.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index c3adc6199..8595ffc32 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -50,7 +50,7 @@ A stack provider may choose to include stack buildpacks with the stack they dist * Is run as the `root` user * Configured with `privileged = true` in the `buildpack.toml` * Can only create one layer -* May include, in the created layer, modifications to any part of the filesystem, excluding ``, ``, and `` directories +* May include, in the created layer, modifications to any part of the filesystem, excluding ``, ``, ``, and `` directories * Does not have access to the application source code during the `build` phase * Is run before all regular buildpacks * Is run against both the build and run images From d4f809a4512051eb67b1bc7dc75333ed9d7b7451 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Sat, 17 Oct 2020 00:46:09 -0500 Subject: [PATCH 086/140] Update text/0000-stack-buildpacks.md Signed-off-by: Joe Kutner Co-authored-by: Emily Casey --- text/0000-stack-buildpacks.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index 8595ffc32..c383ca4d4 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -60,7 +60,7 @@ A stack provider may choose to include stack buildpacks with the stack they dist The stackpack interface is similar to the buildpack interface * The same `bin/detect` and `bin/build` scripts are required * The `bin/detect` will have read-only access to the app -* The positional arguments for `bin/detect` and `bin/build` are the same (though the values may be different) +* The positional arguments for `bin/detect` and `bin/build` are the same * The environment variables and inputs for `bin/detect` and `bin/build` are the same (though the values may be different) However, some of the context it is run in is different from regular buildpacks. From 3ff457c140e9e10a17d5d8766462937ece407fb9 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Sat, 17 Oct 2020 00:47:16 -0500 Subject: [PATCH 087/140] Update text/0000-stack-buildpacks.md Signed-off-by: Joe Kutner Co-authored-by: Emily Casey --- text/0000-stack-buildpacks.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index c383ca4d4..18b1f3552 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -77,7 +77,7 @@ A stack can provide stackpacks by including them in the `/cnb/stack/buildpacks` id = "" ``` -A stackpack will only execute if it passes detection. When the stackpack is executed, its detect and build scripts use the same parameters as the regular buildpacks. But the arguments may differ. For example, the first positional argument to the `bin/build` (the `` directory) MUST NOT be a standard layers directory adhering to the [Buildpacks Build specification](https://github.com/buildpacks/spec/blob/main/buildpack.md#build). +A stackpack will only execute if it passes detection. When the stackpack is executed, its detect and build scripts use the same parameters as the regular buildpacks. The stackpack's snapshot layer may be enriched by writing a `launch.toml` file. The `launch.toml` may define globs of files to be excluded from the image when it is _used_. Any excluded path may also be marked as _cached_, so that those excluded paths are recovered when the image is _used_. The term _used_ is defined as: From 30e3a2b9b1f11ebf95bb2d18e63f8213d4fe89b5 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Sat, 17 Oct 2020 00:47:37 -0500 Subject: [PATCH 088/140] Update text/0000-stack-buildpacks.md Signed-off-by: Joe Kutner Co-authored-by: Emily Casey --- text/0000-stack-buildpacks.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index 18b1f3552..9a5d4c3f5 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -82,7 +82,7 @@ A stackpack will only execute if it passes detection. When the stackpack is exec The stackpack's snapshot layer may be enriched by writing a `launch.toml` file. The `launch.toml` may define globs of files to be excluded from the image when it is _used_. Any excluded path may also be marked as _cached_, so that those excluded paths are recovered when the image is _used_. The term _used_ is defined as: * *Used for build-time build*: A given path is excluded at normal buildpack build-time, and recovered the next time the build image is extended with the stackpack. -* *Used for build-time run*: A given path is excluded from the final image, and recovered the next time the run image is extended with the stackpack (either rebase or rebuild). +* *Used for build-time run*: A given path is excluded from the final image, and restored the next time the run image is extended with the stackpack (either rebase or rebuild). * *Used for rebase run*: A given path is excluded from the rebased image, and recovered the next time the run image is extended with the stackpack (either rebase or rebuild). For example, a stack buildpack may choose to exclude `/var/cache` from the final run image, but may want to mark it as _cached_ to have it restored during build-time and rebase. From bfe7a7d97345176f689d8d5f35232c87d0e8fd83 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Sat, 17 Oct 2020 00:48:10 -0500 Subject: [PATCH 089/140] Update text/0000-stack-buildpacks.md Signed-off-by: Joe Kutner Co-authored-by: Emily Casey --- text/0000-stack-buildpacks.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index 9a5d4c3f5..9fea08bc2 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -161,7 +161,7 @@ During the export phase, the lifecycle will store any snapshot layers created du During the restore phase of the next build, the lifecycle will download the snapshot layer with the cache and store it as a tarball in the `` directory (i.e. it will not extract it). The `restorer` cannot extract the snapshot because it will not run with root privileges. In addition, the `restorer` may run in a different container than the build, which means changes made to the base image are not guaranteed to carry forward. The `builder` will run with root privileges, but will drop privileges after executing stackpacks and before running userspace buildpacks. -During the build phase, the lifecycle will extract and apply each snapshot tarball before running stack buildpacks, but after a snapshot-baseline has been captured. This ensures that all changes from the previous snapshot are preserved even if the stack buildpack does not make any additional changes. +During the build and extend phases, the lifecycle will extract and apply snapshot tarballs before running stack buildpacks, but after a snapshot-baseline has been captured. This ensures that all changes from the previous snapshot are preserved even if the stack buildpack does not make any additional changes. ## Rebasing an App From e4de1eff799ae6aaa3deec69f508b3f1624e829e Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Sat, 17 Oct 2020 00:48:36 -0500 Subject: [PATCH 090/140] Update text/0000-stack-buildpacks.md Signed-off-by: Joe Kutner Co-authored-by: Emily Casey --- text/0000-stack-buildpacks.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index 9fea08bc2..d4ca00028 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -167,7 +167,7 @@ During the build and extend phases, the lifecycle will extract and apply snapsho Before a launch image is rebased, the platform must re-run the any stackpacks that were used to build the launch image against the new run-image. The image containing the stack buildpacks and the builder binary must be provided to the rebaser operation as an argument. A platform may choose to provide the same stack buildpacks and builder binary used during the build that create the launch-image being rebased, or it may provide updates versions (which may increase the risk of something failing in the rebase process). -A platform may choose to store the stack buildpacks and builder binary in any of the following images: +A platform may choose to store the stack buildpacks and extender binary in any of the following images: * A companion image of the launch-image * A builder image that is available when rebasing * The launch-image itself (store the stack buildpacks and builder binary alongside the application itself) From dd0c7bde7d45dd51199a025a74ae99f71d612f3a Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Mon, 19 Oct 2020 09:58:29 -0500 Subject: [PATCH 091/140] Updates from RFC review Signed-off-by: Joe Kutner --- text/0000-stack-buildpacks.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index d4ca00028..3dbf8ecfc 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -51,7 +51,7 @@ A stack provider may choose to include stack buildpacks with the stack they dist * Configured with `privileged = true` in the `buildpack.toml` * Can only create one layer * May include, in the created layer, modifications to any part of the filesystem, excluding ``, ``, ``, and `` directories -* Does not have access to the application source code during the `build` phase +* Must not access the application source code during the `build` phase * Is run before all regular buildpacks * Is run against both the build and run images * Is distributed with the stack run and/or build image @@ -62,12 +62,13 @@ The stackpack interface is similar to the buildpack interface * The `bin/detect` will have read-only access to the app * The positional arguments for `bin/detect` and `bin/build` are the same * The environment variables and inputs for `bin/detect` and `bin/build` are the same (though the values may be different) +* The working directory is `/` instead of the app source directory However, some of the context it is run in is different from regular buildpacks. For each stackpack, the lifecycle will use [snapshotting](https://github.com/GoogleContainerTools/kaniko/blob/master/docs/designdoc.md#snapshotting-snapshotting) to capture changes made during the stackpack's build phase excluding a few specific directories and files. -Additionally, a platform may store a new stack image to cache the changes. All of the captured changes will be included in a single layer produced as output from the stackpack. The `/layers` dir MAY NOT be used to create arbitrary layers. +All of the captured changes will be included in a single layer produced as output from the stackpack. The `/layers` dir MAY NOT be used to create arbitrary layers. A stack can provide stackpacks by including them in the `/cnb/stack/buildpacks` directory, and providing an `/cnb/stack/order.toml` (following the [`order.toml` schema](https://github.com/buildpacks/spec/blob/main/platform.md#ordertoml-toml)) to define their order of execution. The order can be overridden in the `builder.toml` with the following configuration: @@ -79,11 +80,11 @@ A stack can provide stackpacks by including them in the `/cnb/stack/buildpacks` A stackpack will only execute if it passes detection. When the stackpack is executed, its detect and build scripts use the same parameters as the regular buildpacks. -The stackpack's snapshot layer may be enriched by writing a `launch.toml` file. The `launch.toml` may define globs of files to be excluded from the image when it is _used_. Any excluded path may also be marked as _cached_, so that those excluded paths are recovered when the image is _used_. The term _used_ is defined as: +The stackpack's snapshot layer may be enriched by writing a `launch.toml` file. The `launch.toml` may define globs of files to be excluded from the image when it is _exported_. Any excluded path may also be marked as _cached_, so that those excluded paths are recovered when the image is _exported_. The term _exported_ is defined as: -* *Used for build-time build*: A given path is excluded at normal buildpack build-time, and recovered the next time the build image is extended with the stackpack. -* *Used for build-time run*: A given path is excluded from the final image, and restored the next time the run image is extended with the stackpack (either rebase or rebuild). -* *Used for rebase run*: A given path is excluded from the rebased image, and recovered the next time the run image is extended with the stackpack (either rebase or rebuild). +* *Exported for build-time build*: A given path is excluded at userspace buildpack build-time, and recovered the next time the build image is extended with the stackpack. +* *Exported for build-time run*: A given path is excluded from the final image, and restored the next time the run image is extended with the stackpack (either rebase or rebuild). +* *Exported for rebase run*: A given path is excluded from the rebased image, and recovered the next time the run image is extended with the stackpack (either rebase or rebuild). For example, a stack buildpack may choose to exclude `/var/cache` from the final run image, but may want to mark it as _cached_ to have it restored during build-time and rebase. From 5efe0b94a438196f5e12c7dbe7449ef63696da55 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Mon, 19 Oct 2020 10:00:02 -0500 Subject: [PATCH 092/140] Update text/0000-stack-buildpacks.md Signed-off-by: Joe Kutner Co-authored-by: Emily Casey --- text/0000-stack-buildpacks.md | 1 + 1 file changed, 1 insertion(+) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index 3dbf8ecfc..be0e1a82b 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -500,6 +500,7 @@ Usage: | ... | | | | `` | `CNB_STACK_BUILDPACKS_DIR` | `/cnb/stack/buildpacks` | Path to stack buildpacks directory (see [Buildpacks Directory Layout] | `` | `CNB_STACK_GROUP_PATH` | `./stack-group.toml` | Path to output group definition(#buildpacks-directory-layout)) +| ``. | `CNB_STACK_ORDER_PATH`. | `/cnb/stack/order.toml`| Path to order definition (see order.toml) ##### Outputs | Output | Description From b50b2ff35551efc97b1723691c2167919f5c33d7 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Mon, 19 Oct 2020 10:01:27 -0500 Subject: [PATCH 093/140] Update text/0000-stack-buildpacks.md Signed-off-by: Joe Kutner Co-authored-by: Emily Casey --- text/0000-stack-buildpacks.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index be0e1a82b..2aead9b1a 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -491,7 +491,8 @@ Usage: /cnb/lifecycle/detector \ ... [-stack-buildpacks ] \ - [-stack-group ] + [-stack-group ] \ + [-stack-order ] ``` ##### Inputs From 0ee0f37a5437dadacbff26bf0bb5ddbb35984e94 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Mon, 19 Oct 2020 12:37:22 -0500 Subject: [PATCH 094/140] Add provides any to apt stackpack example Signed-off-by: Joe Kutner --- text/0000-stack-buildpacks.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index 2aead9b1a..76a6dd4e7 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -202,6 +202,10 @@ Its `bin/detect` would have the following contents: ```bash #!/usr/bin/env bash +[[provides]] +any = true +mixin = true + exit 0 ``` From ffb45375f08431f59533e5563d211643563f6368 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Mon, 19 Oct 2020 12:44:15 -0500 Subject: [PATCH 095/140] Add stack-layers to exporter Signed-off-by: Joe Kutner --- text/0000-stack-buildpacks.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index 76a6dd4e7..755d79634 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -543,14 +543,16 @@ Usage: ``` /cnb/lifecycle/exporter \ [-stack-group ] \ + [-stack-layers ] \ ... ``` ##### Inputs -| Input | Environment Variable | Default Value | Description -|---------------------|-----------------------|---------------------|--------------------------------------- +| Input | Environment Variable | Default Value | Description +|---------------------|-----------------------------|---------------------------|--------------------------------------- | ... | | | | `` | `CNB_STACK_GROUP_PATH` | `./stack-group.toml` | Path to stack-group file (see [`group.toml`](#grouptoml-toml)) +| `` | `CNB_STACK_LAYERS_DIR` | `/stack-layers` | Path to stack layers directory from extend phase #### `rebaser` From 43a7d1d94226fc79be1eee0457fad648b8928fd5 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Mon, 19 Oct 2020 13:33:10 -0500 Subject: [PATCH 096/140] Update inputs to rebase for stackpacks Signed-off-by: Joe Kutner --- text/0000-stack-buildpacks.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index 755d79634..8fedac67d 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -559,7 +559,8 @@ Usage: Usage: ``` /cnb/lifecycle/rebaser \ - [-build-image ] \ + [-cache-image ] \ + [-plan ] \ ... ``` @@ -567,8 +568,10 @@ Usage: | Input | Environment Variable | Default Value | Description |---------------------|-----------------------|------------------------|--------------------------------------- | ... | | | -| `` | `CNB_BUILD_IMAGE` | derived from `` | Image containing `builder` and stack buildpacks - +| `` | `CNB_CACHE_IMAGE` | | Reference to a cache image in an OCI image registry +| `` | `CNB_PLAN_PATH` | `./plan.toml` | Path to resolved build plan (see [`plan.toml`](#plantoml-toml)) +| `` | `CNB_PLATFORM_DIR` | `/platform` | Path to platform directory +| `` | `CNB_STACK_GROUP_PATH` | `./stack-group.toml` | Path to group definition(#buildpacks-directory-layout)) of buildpacks that run during the extend phase that created the image being rebased. #### `extender` Usage: From b33c1a98e258fd5caaf1ee2a5cedffb75e4fd325 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Mon, 19 Oct 2020 13:38:29 -0500 Subject: [PATCH 097/140] Added details of group and plan used by rebase for stackpacks Signed-off-by: Joe Kutner --- text/0000-stack-buildpacks.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index 8fedac67d..aa24166c3 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -166,7 +166,9 @@ During the build and extend phases, the lifecycle will extract and apply snapsho ## Rebasing an App -Before a launch image is rebased, the platform must re-run the any stackpacks that were used to build the launch image against the new run-image. The image containing the stack buildpacks and the builder binary must be provided to the rebaser operation as an argument. A platform may choose to provide the same stack buildpacks and builder binary used during the build that create the launch-image being rebased, or it may provide updates versions (which may increase the risk of something failing in the rebase process). +Before a launch image is rebased, the platform must re-run the any stackpacks that were used to build the launch image against the new run-image. It will determine which stackpacks to run using a provided [`stack-group.toml`](https://github.com/buildpacks/spec/blob/main/platform.md#grouptoml-toml). To each stackpack, it will pass the build plan derived from the provided [`plan.toml`](https://github.com/buildpacks/spec/blob/main/platform.md#plantoml-toml). + +The image containing the stack buildpacks and the builder binary must be provided to the rebaser operation as an argument. A platform may choose to provide the same stack buildpacks and builder binary used during the build that create the launch-image being rebased, or it may provide updates versions (which may increase the risk of something failing in the rebase process). A platform may choose to store the stack buildpacks and extender binary in any of the following images: * A companion image of the launch-image From 0d9824a4b45d23da1727101a08fdf509c31e94f7 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Thu, 22 Oct 2020 01:14:58 -0500 Subject: [PATCH 098/140] Remove dynamically provided mixins from the build plan Signed-off-by: Joe Kutner --- text/0000-stack-buildpacks.md | 23 +---------------------- 1 file changed, 1 insertion(+), 22 deletions(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index aa24166c3..6b82b1332 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -92,12 +92,7 @@ For example, a stack buildpack may choose to exclude `/var/cache` from the final ### Providing Mixins -A stack buildpack MAY define a set of mixins it provides in either of two ways: - -1. Statically in the `buildpack.toml` -1. Dynamically in the Build Plan. - -A stack buildpack MAY define a set of stack specific mixins in the `buildpack.toml` with the following schema: +A stack buildpack MAY define a set of mixins it provides statically in the `buildpack.toml` with the following schema: ``` [[stacks]] @@ -108,15 +103,6 @@ any = mixins = [ "mixin name" ] ``` -Additionally, mixins MAY be dynamically provided in the build plan: - -``` -[[provides]] -name = "" -any = -mixin = -``` - A userspace buildpack MAY require mixins in the build plan ``` @@ -136,7 +122,6 @@ Stack buildpacks MUST NOT require any entries in the build plan (neither mixins After the detect phase, the lifecycle will merge the list of provided mixins from the following sources: * `stack.toml` * `buildpack.toml` of any stack buildpacks -* The build plan (any `[[provides]]` tables with `mixin = true`) If any required mixins from the Build Plan (any `[[required]]` tables with `mixin = true`) are not provided, then the build will fail. Otherwise the build will continue. @@ -204,10 +189,6 @@ Its `bin/detect` would have the following contents: ```bash #!/usr/bin/env bash -[[provides]] -any = true -mixin = true - exit 0 ``` @@ -451,7 +432,6 @@ Under the `[stacks.provides]` table: ``` [[provides]] name = "" -mixin = [[requires]] name = "" @@ -464,7 +444,6 @@ mixin = [[or.provides]] name = "" -mixin = [[or.requires]] name = "" From d8e07fc4305177a314d7a07d5c316e2891afdb11 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Thu, 22 Oct 2020 01:31:16 -0500 Subject: [PATCH 099/140] Fix buildpack.toml schema Signed-off-by: Joe Kutner --- text/0000-stack-buildpacks.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index aa24166c3..a61bb17d5 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -420,7 +420,7 @@ Where: ## buildpack.toml (TOML) - This proposal adds new keys to the `[buildpack]` table in `buildpack.toml`, and a new `[[mixins]]` array of tables: + This proposal adds new keys to the `[buildpack]` table in `buildpack.toml`, and a new `mixins` array: ``` [buildpack] From 233e80dc276ef724e2027d522b3cd58097136137 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Sat, 24 Oct 2020 09:05:03 -0500 Subject: [PATCH 100/140] Address working group feedback Signed-off-by: Joe Kutner --- text/0000-stack-buildpacks.md | 20 +++++++++++++------- 1 file changed, 13 insertions(+), 7 deletions(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index 270e01676..92b070e17 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -66,9 +66,11 @@ The stackpack interface is similar to the buildpack interface However, some of the context it is run in is different from regular buildpacks. -For each stackpack, the lifecycle will use [snapshotting](https://github.com/GoogleContainerTools/kaniko/blob/master/docs/designdoc.md#snapshotting-snapshotting) to capture changes made during the stackpack's build phase excluding a few specific directories and files. +For each stackpack, the lifecycle will use [snapshotting](https://github.com/GoogleContainerTools/kaniko/blob/master/docs/designdoc.md#snapshotting-snapshotting) to capture changes made during the stackpack's build or extend phases excluding a few specific directories and files. -All of the captured changes will be included in a single layer produced as output from the stackpack. The `/layers` dir MAY NOT be used to create arbitrary layers. +All of the captured changes in each phase (build or extend) will be included in a single layer produced as output from the stackpack. During the build phase, the snapshot will be stored in the `/layers` directory. During the extend phase, the snapshot will be stored in the `/run-layers` directory. + +The `/layers` dir MAY NOT be used to create arbitrary layers. A stack can provide stackpacks by including them in the `/cnb/stack/buildpacks` directory, and providing an `/cnb/stack/order.toml` (following the [`order.toml` schema](https://github.com/buildpacks/spec/blob/main/platform.md#ordertoml-toml)) to define their order of execution. The order can be overridden in the `builder.toml` with the following configuration: @@ -80,7 +82,7 @@ A stack can provide stackpacks by including them in the `/cnb/stack/buildpacks` A stackpack will only execute if it passes detection. When the stackpack is executed, its detect and build scripts use the same parameters as the regular buildpacks. -The stackpack's snapshot layer may be enriched by writing a `launch.toml` file. The `launch.toml` may define globs of files to be excluded from the image when it is _exported_. Any excluded path may also be marked as _cached_, so that those excluded paths are recovered when the image is _exported_. The term _exported_ is defined as: +The stackpack's snapshot layer may be enriched by writing a `stack-layer.toml` file. The `stack-layer.toml` may define globs of files to be excluded from the image when it is _exported_. Any excluded path may also be marked as _cached_, so that those excluded paths are recovered when the image is _exported_. The term _exported_ is defined as: * *Exported for build-time build*: A given path is excluded at userspace buildpack build-time, and recovered the next time the build image is extended with the stackpack. * *Exported for build-time run*: A given path is excluded from the final image, and restored the next time the run image is extended with the stackpack (either rebase or rebuild). @@ -145,13 +147,15 @@ A Stack Buildpack that needs to install mixins must select them from the build p During the export phase, the lifecycle will store any snapshot layers created during the build phase in the cache. -During the restore phase of the next build, the lifecycle will download the snapshot layer with the cache and store it as a tarball in the `` directory (i.e. it will not extract it). The `restorer` cannot extract the snapshot because it will not run with root privileges. In addition, the `restorer` may run in a different container than the build, which means changes made to the base image are not guaranteed to carry forward. The `builder` will run with root privileges, but will drop privileges after executing stackpacks and before running userspace buildpacks. +During the restore phase of the next build, the lifecycle will download the snapshot layer with the cache and store it as a tarball in the `` directory and the `` directory (i.e. it will not extract it). The `restorer` cannot extract the snapshot because it will not run with root privileges. In addition, the `restorer` may run in a different container than the build, which means changes made to the base image are not guaranteed to carry forward. The `builder` will run with root privileges and untar the snapshot and cache, but will drop privileges after executing stackpacks and before running userspace buildpacks. The `extender` will also run with root privileges and untar the cache. During the build and extend phases, the lifecycle will extract and apply snapshot tarballs before running stack buildpacks, but after a snapshot-baseline has been captured. This ensures that all changes from the previous snapshot are preserved even if the stack buildpack does not make any additional changes. +At the end of the build phase, the lifeyclce will forcefully delete everything not in `[[excludes]]` after ALL stackpacks run, before buildpacks run (which implies implies that it does not need to do this for extend phase). + ## Rebasing an App -Before a launch image is rebased, the platform must re-run the any stackpacks that were used to build the launch image against the new run-image. It will determine which stackpacks to run using a provided [`stack-group.toml`](https://github.com/buildpacks/spec/blob/main/platform.md#grouptoml-toml). To each stackpack, it will pass the build plan derived from the provided [`plan.toml`](https://github.com/buildpacks/spec/blob/main/platform.md#plantoml-toml). +Before a launch image is rebased, the platform must re-run the any stackpacks that were used to build the launch image against the new run-image. It will determine which stackpacks to run using a provided [`stack-group.toml`](https://github.com/buildpacks/spec/blob/main/platform.md#grouptoml-toml). The Build Plan will be stored in a LABEL of the launch-image, which will be serialized back to a `plan.toml` during rebase. To each stackpack, it will pass the build plan derived from the provided [`plan.toml`](https://github.com/buildpacks/spec/blob/main/platform.md#plantoml-toml). The image containing the stack buildpacks and the builder binary must be provided to the rebaser operation as an argument. A platform may choose to provide the same stack buildpacks and builder binary used during the build that create the launch-image being rebased, or it may provide updates versions (which may increase the risk of something failing in the rebase process). @@ -203,7 +207,7 @@ for package in $(cat $3 | yj -t | jq -r ".entries | .[] | .name"); do apt install $package done -cat << EOF > $1/launch.toml +cat << EOF > $1/stack-layer.toml [[excludes]] paths = [ "/var/cache" ] cache = true @@ -382,18 +386,20 @@ Stack buildpacks are identical to other buildpacks, with the following exception * `/etc/hostname`, `/etc/hosts`, `/etc/mtab`, `/etc/resolv.conf` * `/.dockerenv` -## launch.toml (TOML) +## stack-layer.toml (TOML) ``` [[excludes]] paths = [""] cache = false +restore = false ``` Where: * `paths` = a list of paths to exclude from the layer * `cache` = if true, the paths will be excluded from the launch image layer, but will be included in the cache layer. +* `restore` = if true, the paths will be restored during the next build. 1. Paths not referenced by an `[[excludes]]` entry will be included in the cache _and_ run image (default). 1. Any paths with an `[[excludes]]` entry and `cache = true` will be included in the cache image, but not the run image. From f6c832d0852081b226f9e48f13ef1e2df3f9efda Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Mon, 26 Oct 2020 08:27:09 -0500 Subject: [PATCH 101/140] Clean up Signed-off-by: Joe Kutner --- text/0000-stack-buildpacks.md | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index 92b070e17..aec59fdd5 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -57,15 +57,13 @@ A stack provider may choose to include stack buildpacks with the stack they dist * Is distributed with the stack run and/or build image * May not create layers using the `` directory -The stackpack interface is similar to the buildpack interface +The stackpack interface is similar to the buildpack interface: * The same `bin/detect` and `bin/build` scripts are required * The `bin/detect` will have read-only access to the app * The positional arguments for `bin/detect` and `bin/build` are the same * The environment variables and inputs for `bin/detect` and `bin/build` are the same (though the values may be different) * The working directory is `/` instead of the app source directory -However, some of the context it is run in is different from regular buildpacks. - For each stackpack, the lifecycle will use [snapshotting](https://github.com/GoogleContainerTools/kaniko/blob/master/docs/designdoc.md#snapshotting-snapshotting) to capture changes made during the stackpack's build or extend phases excluding a few specific directories and files. All of the captured changes in each phase (build or extend) will be included in a single layer produced as output from the stackpack. During the build phase, the snapshot will be stored in the `/layers` directory. During the extend phase, the snapshot will be stored in the `/run-layers` directory. @@ -105,6 +103,10 @@ any = mixins = [ "mixin name" ] ``` +### Requiring Mixins + +A stack buildpack MAY NOT require any entries in the build plan (neither mixins nor non-mixins). This ensures that we do not need to re-run the detect phase. + A userspace buildpack MAY require mixins in the build plan ``` @@ -115,17 +117,13 @@ mixin = A userspace buildpack MAY NOT provide mixins in the build plan. -### Requiring Mixins - -Stack buildpacks MUST NOT require any entries in the build plan (neither mixins nor non-mixins). This ensures that we do not need to re-run the detect phase. - ### Resolving Mixins After the detect phase, the lifecycle will merge the list of provided mixins from the following sources: * `stack.toml` * `buildpack.toml` of any stack buildpacks -If any required mixins from the Build Plan (any `[[required]]` tables with `mixin = true`) are not provided, then the build will fail. Otherwise the build will continue. +If any required mixins from the Build Plan (any `[[required]]` tables with `mixins`) are not provided, then the build will fail. Otherwise the build will continue. If a stack buildpack provides a mixin that is not required, the stack buildpack MAY pass detection. This is in contrast to a userspace buildpack providing a dependency that is not required, which fails detection. If a mixin is required for a [single stage only](https://github.com/buildpacks/rfcs/pull/109) with the `build:` or `run:` prefix, a stack buildpack may provide it for both stages without failing detection. However, it will not be included in the Buildpack Build Plan during the stage where it is not required. From e1f5c18fb14b9a40c3edc0958f9955672bf04d61 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Wed, 28 Oct 2020 09:19:17 -0500 Subject: [PATCH 102/140] Update to mixins in the build plan Signed-off-by: Joe Kutner --- text/0000-stack-buildpacks.md | 22 +++++++++------------- 1 file changed, 9 insertions(+), 13 deletions(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index aec59fdd5..4ffeaebbd 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -135,8 +135,7 @@ The entries of the build plan during the build and extend phases will have a new ``` [[entries]] -name = "libpq" -mixin = true +mixin = "libpq" ``` A Stack Buildpack that needs to install mixins must select them from the build plan. @@ -201,7 +200,7 @@ Its `bin/build` would have the following contents: apt update -y -for package in $(cat $3 | yj -t | jq -r ".entries | .[] | .name"); do +for package in $(cat $3 | yj -t | jq -r ".entries | .[] | .mixin"); do apt install $package done @@ -410,7 +409,6 @@ Where: ``` [buildpack] privileged = -non-idempotent = [[stacks]] id = "" @@ -419,17 +417,15 @@ id = "" mixins = [ "mixin name" ] [stacks.provides] -any = -mixins = [ "mixin name" ] +mixins = [ "mixin name or *" ] ``` * `privileged` - when set to `true`, the lifecycle will run this buildpack as the `root` user. -* `non-idempotent` - when set to `true`, indicates that the buildpack is not idempotent. The lifecycle will provide a clean filesystem from the stack image(s) before each run (i.e. no cache). Under the `[stacks.provides]` table: * `any` - a boolean that, when true, indicates that the buildpack can provide all mixins -* `mixins` - a list of names that match mixins provided by this buildpack +* `mixins` - a list of names that match mixins provided by this buildpack, or the `*` representing all mixins. ## Build Plan (TOML) @@ -439,10 +435,10 @@ name = "" [[requires]] name = "" -mixin = +mixins = [ "" ] [requires.metadata] -# buildpack-specific data +# buildpack-specific data; not allowed for mixins [[or]] @@ -451,10 +447,10 @@ name = "" [[or.requires]] name = "" -mixin = +mixins = [ "" ] [or.requires.metadata] -# buildpack-specific data +# buildpack-specific data; not allowed for mixins ``` ### Buildpack Plan (TOML) @@ -462,7 +458,7 @@ mixin = ``` [[entries]] name = "" -mixin = +mixin = "" [entries.metadata] # buildpack-specific data From 7ad2c0903317517d73627dd67e2444218acff0f1 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Wed, 28 Oct 2020 13:03:06 -0500 Subject: [PATCH 103/140] Merged in changes from branch stackpacks-sl-jb Signed-off-by: Joe Kutner --- text/0000-stack-buildpacks.md | 36 +++++++++++++++++------------------ 1 file changed, 18 insertions(+), 18 deletions(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index 4ffeaebbd..71c65dd2a 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -18,7 +18,7 @@ This is a proposal for a new type of buildpack that runs against a stack in orde Normally, buildpacks do not run as root. This is an intentional design decision that ensures operations like `rebase` will work on day-2. -However, many applications and buildpacks require modifications to the stack they run on, such as adding system packages or custom certificates. For this reason, we need a mechanism that buildpack authors and buildpack users can leverage to extend their stacks. +However, many applications and buildpacks require modifications to the stack they run on, such as adding system packages or custom certificates. For this reason, we need a mechanism that stack authors, buildpack authors, and buildpack users can leverage to extend their stacks. # What it is [what-it-is]: #what-it-is @@ -28,18 +28,18 @@ However, many applications and buildpacks require modifications to the stack the - *userspace buildpack* - the traditional definition of a buildpack (i.e. does not run as root, and runs against an app) - *launch image* - the final image produced by a buildpack build. It consists of a run-image combined with layers created by buildpacks. -A new type of buildpack, called a Stack buildpack, may run against a stack (both build and run images) in order to extend it in ways that are only possible by running privileged commands. Unlike userspace buildpacks, stack buildpack can modify any path on the filesystem. Userspace buildpack can only create/modify disjoint layers (either by adding a dir to `` or modifying an app slice), which makes possible features like individual layer reuse that is independent or ordering. +A new type of buildpack, called a stack buildpack, may run against a stack (both build and run images) in order to extend it in ways that are only possible by running privileged commands. Unlike userspace buildpacks, stack buildpack can modify any path on the filesystem. Userspace buildpack can only create/modify disjoint layers (either by adding a dir to `` or modifying an app slice), which makes possible features like individual layer reuse that is independent or ordering. -A stackpack may also define a list of mixins that it provides to the stack, or indicate that it will provide _any_ mixin. In this way, a stack that is missing a mixin required by a buildpack may have that mixin provided by a stack buildpack. +A stackpack may also define a list of mixins that it can provide to the stack, or indicate that it can provide _any_ mixin. In this way, a stack that is missing a mixin required by a buildpack may have that mixin provided by a stack buildpack. -A stack provider may choose to include stack buildpacks with the stack they distribute. If a stack includes any stack buildpacks, the following will occur when the build phase starts: +A stack provider may choose to include stack buildpacks with the stack they distribute. If a stack includes a `/cnb/stack/order.toml` file and associated stackpacks, then the following will occur: -1. The platform may compare the list of requested mixins with the static list of mixins provided by the stack buildpacks, and fail the build if it chooses to do so. -1. The lifecycle will compare the list of required mixins to the list of mixins provided by stack and stack buildpacks in accordance with [stage-specific mixin rules](https://github.com/buildpacks/rfcs/pull/109). If any mixins are still not provided, the build will fail. -1. The lifecycle will run the detect phase for all stackpacks defined in the build-image. -1. The lifecycle will execute the stack buildpack build phase for all passing stackpack(s). If no stack buildpacks pass detect, the build will continue the build phase as normal (running userspace buildpacks). -1. After, during, or before the lifecycle's build phase, the lifecycle will begin a new phase called _extend_. -1. During the extend phase, stack buildpacks that passed detection will run against the run images accordingly (see details below). +* Before any lifecycle phases: + 1. The platform may compare the list of mixins that are statically required by all buildpacks (in the `stacks` sections of their `buildpack.toml` files) with the static list of mixins provided by the stack buildpacks (in the `stacks` sections of their `buildpack.toml` files), and fail the build if it chooses to do so. + +* During the detect phase: + 1. The lifecycle will compare the list of required mixins to the list of mixins provided by stack and stack buildpacks in accordance with [stage-specific mixin rules](https://github.com/buildpacks/rfcs/pull/109). If any mixins are still not provided, the build will fail. To accomplish this, the list of run-time and build-time mixins that are already present in the run image and build image must be provided to the detector. + 1. The lifecycle will run the detect phase for all stackpacks defined in the `/cnb/stack/order.toml`, and then for all userspace buildpacks defined in `/cnb/order.toml`. # How it Works @@ -49,7 +49,7 @@ A stack provider may choose to include stack buildpacks with the stack they dist * Is run as the `root` user * Configured with `privileged = true` in the `buildpack.toml` -* Can only create one layer +* Can only create one layer per image (both a layer for buildpacks to build on top of, and a layer for the app image) * May include, in the created layer, modifications to any part of the filesystem, excluding ``, ``, ``, and `` directories * Must not access the application source code during the `build` phase * Is run before all regular buildpacks @@ -59,14 +59,14 @@ A stack provider may choose to include stack buildpacks with the stack they dist The stackpack interface is similar to the buildpack interface: * The same `bin/detect` and `bin/build` scripts are required -* The `bin/detect` will have read-only access to the app +* The `bin/detect` script must not write to the app * The positional arguments for `bin/detect` and `bin/build` are the same * The environment variables and inputs for `bin/detect` and `bin/build` are the same (though the values may be different) * The working directory is `/` instead of the app source directory For each stackpack, the lifecycle will use [snapshotting](https://github.com/GoogleContainerTools/kaniko/blob/master/docs/designdoc.md#snapshotting-snapshotting) to capture changes made during the stackpack's build or extend phases excluding a few specific directories and files. -All of the captured changes in each phase (build or extend) will be included in a single layer produced as output from the stackpack. During the build phase, the snapshot will be stored in the `/layers` directory. During the extend phase, the snapshot will be stored in the `/run-layers` directory. +All of the captured changes in each phase (build or extend) will be included in a single layer (one for run, one for build) produced as output from the stackpack. During the build phase, the snapshot will be stored in the `/layers` directory. During the extend phase, the snapshot will be stored in the `/run-layers` directory. The `/layers` dir MAY NOT be used to create arbitrary layers. @@ -99,8 +99,7 @@ A stack buildpack MAY define a set of mixins it provides statically in the `buil id = "" [stacks.provides] -any = -mixins = [ "mixin name" ] +mixins = [ "" ] ``` ### Requiring Mixins @@ -112,7 +111,7 @@ A userspace buildpack MAY require mixins in the build plan ``` [[requires]] name = "" -mixin = +mixins = [ "" ] ``` A userspace buildpack MAY NOT provide mixins in the build plan. @@ -121,6 +120,7 @@ A userspace buildpack MAY NOT provide mixins in the build plan. After the detect phase, the lifecycle will merge the list of provided mixins from the following sources: * `stack.toml` +* `run-stack.toml` (for extend stage resolution) * `buildpack.toml` of any stack buildpacks If any required mixins from the Build Plan (any `[[required]]` tables with `mixins`) are not provided, then the build will fail. Otherwise the build will continue. @@ -414,10 +414,10 @@ privileged = id = "" [stacks.requires] -mixins = [ "mixin name" ] +mixins = [ "" ] [stacks.provides] -mixins = [ "mixin name or *" ] +mixins = [ "" ] ``` * `privileged` - when set to `true`, the lifecycle will run this buildpack as the `root` user. From f12ed2eb0dc978172864dacb2bb6dc6d40061e05 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Wed, 28 Oct 2020 14:07:00 -0500 Subject: [PATCH 104/140] Update text/0000-stack-buildpacks.md Signed-off-by: Joe Kutner Co-authored-by: Emily Casey --- text/0000-stack-buildpacks.md | 1 - 1 file changed, 1 deletion(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index 71c65dd2a..aaffd55de 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -424,7 +424,6 @@ mixins = [ "" ] Under the `[stacks.provides]` table: -* `any` - a boolean that, when true, indicates that the buildpack can provide all mixins * `mixins` - a list of names that match mixins provided by this buildpack, or the `*` representing all mixins. ## Build Plan (TOML) From 82456228e4312a57e2282f1e2003102a99a721aa Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Wed, 28 Oct 2020 14:07:12 -0500 Subject: [PATCH 105/140] Update text/0000-stack-buildpacks.md Signed-off-by: Joe Kutner Co-authored-by: Emily Casey --- text/0000-stack-buildpacks.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index aaffd55de..1df269ba9 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -395,7 +395,7 @@ restore = false Where: * `paths` = a list of paths to exclude from the layer -* `cache` = if true, the paths will be excluded from the launch image layer, but will be included in the cache layer. +* `cache` = if true, the paths will be excluded from the launch image layer, but will be included in a cached layer. * `restore` = if true, the paths will be restored during the next build. 1. Paths not referenced by an `[[excludes]]` entry will be included in the cache _and_ run image (default). From eb9cb5d00caafce755e011f40e760cab5bc05649 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Wed, 28 Oct 2020 14:11:36 -0500 Subject: [PATCH 106/140] Added stack-layer.toml as output to spec changes Signed-off-by: Joe Kutner --- text/0000-stack-buildpacks.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index 1df269ba9..ae9dcc040 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -517,6 +517,7 @@ Usage: |--------------------------------------------|---------------------------------------------- | ... | | `/.tgz` | Layer snapshot (see [Buildpack Interface Specfication](buildpack.md) +| `/stack-layer.toml` | Layer snaphot metadata #### `exporter` Usage: @@ -582,6 +583,7 @@ Usage: | `/dev/stdout` | Logs (info) | `/dev/stderr` | Logs (warnings, errors) | `/.tgz` | Layer snapshot (see [Buildpack Interface Specfication](buildpack.md) +| `/stack-layer.toml` | Layer snaphot metadata | `/config/metadata.toml` | Build metadata (see [`metadata.toml`](#metadatatoml-toml)) | Exit Code | Result| From b9cf2b1dfcf2d1235fc6750778b97ee301a3d1bb Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Thu, 29 Oct 2020 09:23:44 -0500 Subject: [PATCH 107/140] Update description of exclude cache Signed-off-by: Joe Kutner --- text/0000-stack-buildpacks.md | 13 +++++-------- 1 file changed, 5 insertions(+), 8 deletions(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index ae9dcc040..e5dd89903 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -110,8 +110,7 @@ A userspace buildpack MAY require mixins in the build plan ``` [[requires]] -name = "" -mixins = [ "" ] +mixin = "" ``` A userspace buildpack MAY NOT provide mixins in the build plan. @@ -389,14 +388,12 @@ Stack buildpacks are identical to other buildpacks, with the following exception [[excludes]] paths = [""] cache = false -restore = false ``` Where: -* `paths` = a list of paths to exclude from the layer -* `cache` = if true, the paths will be excluded from the launch image layer, but will be included in a cached layer. -* `restore` = if true, the paths will be restored during the next build. +* `paths` = a list of paths to exclude from the launch image layer in the extend phase. During the build phase, these paths will be removed from the filesystem before executing any userspace buildpacks. +* `cache` = if true, the paths will be cached even if they are removed from the layer. 1. Paths not referenced by an `[[excludes]]` entry will be included in the cache _and_ run image (default). 1. Any paths with an `[[excludes]]` entry and `cache = true` will be included in the cache image, but not the run image. @@ -434,7 +431,7 @@ name = "" [[requires]] name = "" -mixins = [ "" ] +mixin = "" [requires.metadata] # buildpack-specific data; not allowed for mixins @@ -446,7 +443,7 @@ name = "" [[or.requires]] name = "" -mixins = [ "" ] +mixin = "" [or.requires.metadata] # buildpack-specific data; not allowed for mixins From 12d773226480631b78cacb214201b7d467467ab7 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Thu, 29 Oct 2020 13:04:50 -0500 Subject: [PATCH 108/140] fix typos Signed-off-by: Joe Kutner --- text/0000-stack-buildpacks.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index e5dd89903..bd9f6c5e0 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -147,7 +147,7 @@ During the restore phase of the next build, the lifecycle will download the snap During the build and extend phases, the lifecycle will extract and apply snapshot tarballs before running stack buildpacks, but after a snapshot-baseline has been captured. This ensures that all changes from the previous snapshot are preserved even if the stack buildpack does not make any additional changes. -At the end of the build phase, the lifeyclce will forcefully delete everything not in `[[excludes]]` after ALL stackpacks run, before buildpacks run (which implies implies that it does not need to do this for extend phase). +At the end of the build phase, the lifecycle will forcefully delete everything not in `[[excludes]]` after ALL stackpacks run, before buildpacks run (which implies that it does not need to do this for extend phase). ## Rebasing an App From 7990cc1925bfecfe598b4ef716b2dfe998168364 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Thu, 29 Oct 2020 14:11:52 -0500 Subject: [PATCH 109/140] Update 0000-stack-buildpacks.md Signed-off-by: Joe Kutner --- text/0000-stack-buildpacks.md | 69 ++++++++++++++++++++--------------- 1 file changed, 40 insertions(+), 29 deletions(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index bd9f6c5e0..60f5c64b1 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -23,12 +23,12 @@ However, many applications and buildpacks require modifications to the stack the # What it is [what-it-is]: #what-it-is -- *root buildpack* - a new type of buildpack that runs as the root user -- *stack buildpack* - (a.k.a. stackpack) a type of root buildpack that runs against the stack image(s) instead of an app. Stack buildpacks must not make changes to the build and run images that either violate stack [compatibility guarantees](https://github.com/buildpacks/spec/blob/main/platform.md#compatibility-guarantees) or violate the contract defined by that stack's author. +- *stack buildpack* - (a.k.a. stackpack) a type of buildpack that runs with root privileges against the stack image(s) instead of an app. Stack buildpacks must not make changes to the build and run images that either violate stack [compatibility guarantees](https://github.com/buildpacks/spec/blob/main/platform.md#compatibility-guarantees) or violate the contract defined by that stack's author. - *userspace buildpack* - the traditional definition of a buildpack (i.e. does not run as root, and runs against an app) -- *launch image* - the final image produced by a buildpack build. It consists of a run-image combined with layers created by buildpacks. +- *app image* - the final image produced by a buildpack build. It consists of a run-image combined with layers created by buildpacks. +- *extend phase* - a new buildpack phase in which a stack buildpack may create a layer for the app image. -A new type of buildpack, called a stack buildpack, may run against a stack (both build and run images) in order to extend it in ways that are only possible by running privileged commands. Unlike userspace buildpacks, stack buildpack can modify any path on the filesystem. Userspace buildpack can only create/modify disjoint layers (either by adding a dir to `` or modifying an app slice), which makes possible features like individual layer reuse that is independent or ordering. +A new type of buildpack, called a stack buildpack, may run against a stack (both build and run images) in order to extend it in ways that are only possible by running privileged commands. Unlike userspace buildpacks, stack buildpack can modify any path (with exceptions) on the filesystem. Userspace buildpack can only create/modify disjoint layers (either by adding a dir to `` or modifying an app slice), which makes possible features like individual layer reuse that is independent or ordering. A stackpack may also define a list of mixins that it can provide to the stack, or indicate that it can provide _any_ mixin. In this way, a stack that is missing a mixin required by a buildpack may have that mixin provided by a stack buildpack. @@ -39,8 +39,14 @@ A stack provider may choose to include stack buildpacks with the stack they dist * During the detect phase: 1. The lifecycle will compare the list of required mixins to the list of mixins provided by stack and stack buildpacks in accordance with [stage-specific mixin rules](https://github.com/buildpacks/rfcs/pull/109). If any mixins are still not provided, the build will fail. To accomplish this, the list of run-time and build-time mixins that are already present in the run image and build image must be provided to the detector. - 1. The lifecycle will run the detect phase for all stackpacks defined in the `/cnb/stack/order.toml`, and then for all userspace buildpacks defined in `/cnb/order.toml`. + 1. The lifecycle will merge and run the detect phase for all stackpacks defined in the `/cnb/stack/order.toml` and for all userspace buildpacks defined in `/cnb/order.toml`. +* During the build phase (potentially in parallel to extend phase): + 1. The lifecycle will execute the stack buildpack build phase for all passing stackpack(s) as root. + 1. The lifecycle will drop privilidges and continue the build phase as normal (running userspace buildpacks). + +* During the extend phase (potentially in parallel to build phase): + 1. During the extend phase, stack buildpacks that passed detection will run against the run images accordingly (see details below). # How it Works [how-it-works]: #how-it-works @@ -49,26 +55,25 @@ A stack provider may choose to include stack buildpacks with the stack they dist * Is run as the `root` user * Configured with `privileged = true` in the `buildpack.toml` -* Can only create one layer per image (both a layer for buildpacks to build on top of, and a layer for the app image) -* May include, in the created layer, modifications to any part of the filesystem, excluding ``, ``, ``, and `` directories +* Can only create one layer that is exported in the app image +* May include, in the created layer, modifications to any part of the filesystem, excluding ``, ``, ``, and `` directories as well as other directories [listed below](#spec-changes-optional). * Must not access the application source code during the `build` phase -* Is run before all regular buildpacks -* Is run against both the build and run images -* Is distributed with the stack run and/or build image +* Must run before all userspace buildpacks if it passes detection +* Must run against both the build and run images if it passes detection for that stack type +* Must be distributed with the stack build image, and may be distributed with the stack run image * May not create layers using the `` directory The stackpack interface is similar to the buildpack interface: * The same `bin/detect` and `bin/build` scripts are required -* The `bin/detect` script must not write to the app -* The positional arguments for `bin/detect` and `bin/build` are the same -* The environment variables and inputs for `bin/detect` and `bin/build` are the same (though the values may be different) +* The positional arguments for `bin/detect` and `bin/build` are the same as with userspace buildpacks +* The environment variables and inputs for `bin/detect` and `bin/build` are the same as with userspace buildpacks (though the values may be different) * The working directory is `/` instead of the app source directory For each stackpack, the lifecycle will use [snapshotting](https://github.com/GoogleContainerTools/kaniko/blob/master/docs/designdoc.md#snapshotting-snapshotting) to capture changes made during the stackpack's build or extend phases excluding a few specific directories and files. -All of the captured changes in each phase (build or extend) will be included in a single layer (one for run, one for build) produced as output from the stackpack. During the build phase, the snapshot will be stored in the `/layers` directory. During the extend phase, the snapshot will be stored in the `/run-layers` directory. +All of the captured changes in the extend phase will be included in a single layer produced as output from the stackpack, which will be mounted into the `/run-layers` directory of the export container. Any changes performed by the stack buildpack to the build image will persist through execution of userspace buildpacks, but not exported as a layer. -The `/layers` dir MAY NOT be used to create arbitrary layers. +The `` dir MAY NOT be used to create arbitrary layers. A stack can provide stackpacks by including them in the `/cnb/stack/buildpacks` directory, and providing an `/cnb/stack/order.toml` (following the [`order.toml` schema](https://github.com/buildpacks/spec/blob/main/platform.md#ordertoml-toml)) to define their order of execution. The order can be overridden in the `builder.toml` with the following configuration: @@ -78,15 +83,15 @@ A stack can provide stackpacks by including them in the `/cnb/stack/buildpacks` id = "" ``` -A stackpack will only execute if it passes detection. When the stackpack is executed, its detect and build scripts use the same parameters as the regular buildpacks. +A stackpack will only execute if it passes detection. When the stackpack is executed, its detect and build scripts use the same parameters as the userspace buildpacks. -The stackpack's snapshot layer may be enriched by writing a `stack-layer.toml` file. The `stack-layer.toml` may define globs of files to be excluded from the image when it is _exported_. Any excluded path may also be marked as _cached_, so that those excluded paths are recovered when the image is _exported_. The term _exported_ is defined as: +The stackpack's snapshot layer may be enriched by writing a `stack-layer.toml` file. The `stack-layer.toml` may define globs of files to be excluded from the image when it is _exported_. Any excluded path may also be marked as _cached_, so that those excluded paths are recovered before the build or extend phase. The term _exported_ is defined as: * *Exported for build-time build*: A given path is excluded at userspace buildpack build-time, and recovered the next time the build image is extended with the stackpack. * *Exported for build-time run*: A given path is excluded from the final image, and restored the next time the run image is extended with the stackpack (either rebase or rebuild). * *Exported for rebase run*: A given path is excluded from the rebased image, and recovered the next time the run image is extended with the stackpack (either rebase or rebuild). -For example, a stack buildpack may choose to exclude `/var/cache` from the final run image, but may want to mark it as _cached_ to have it restored during build-time and rebase. +For example, a stack buildpack may choose to exclude `/var/cache` from the final app image, but may want to mark it as _cached_ to have it restored before the extend phase. ## Mixins @@ -118,13 +123,19 @@ A userspace buildpack MAY NOT provide mixins in the build plan. ### Resolving Mixins After the detect phase, the lifecycle will merge the list of provided mixins from the following sources: -* `stack.toml` -* `run-stack.toml` (for extend stage resolution) +* Build stack image mixin list +* Run stack image mixin list * `buildpack.toml` of any stack buildpacks -If any required mixins from the Build Plan (any `[[required]]` tables with `mixins`) are not provided, then the build will fail. Otherwise the build will continue. +If any required mixins from the Build Plan (any `[[requires]]` tables with `mixins`) are not provided, then the build will fail. Otherwise the build will continue. + +If a stack buildpack provides a mixin that is not required, the stack buildpack MAY pass detection. For each phase, if a stackpack: +* provides mixins, and at least one of those mixins are required; it MAY pass +* provides mixins, and none of those mixins are required; it MUST be skipped +* provides mixins, and none of those mixins are required, but it also provides another dependency (non-mixin), which is required; it MAY pass +* does not provide mixins; it MAY pass -If a stack buildpack provides a mixin that is not required, the stack buildpack MAY pass detection. This is in contrast to a userspace buildpack providing a dependency that is not required, which fails detection. If a mixin is required for a [single stage only](https://github.com/buildpacks/rfcs/pull/109) with the `build:` or `run:` prefix, a stack buildpack may provide it for both stages without failing detection. However, it will not be included in the Buildpack Build Plan during the stage where it is not required. +This is in contrast to a userspace buildpack providing a dependency that is not required, which fails detection. If a mixin is required for a [single stage only](https://github.com/buildpacks/rfcs/pull/109) with the `build:` or `run:` prefix, a stack buildpack may provide it for both stages without failing detection. However, it will not be included in the Buildpack Build Plan during the stage where it is not required. During the build phase, the lifecycle will create a build plan containing only the entries required during that stage (build or run). * If a mixin is required for "run" stage only, it will not appear in the build plan entries during build @@ -137,7 +148,7 @@ The entries of the build plan during the build and extend phases will have a new mixin = "libpq" ``` -A Stack Buildpack that needs to install mixins must select them from the build plan. +A stack buildpack that needs to install mixins must select them from the build plan. ## Caching and Restoring @@ -151,20 +162,20 @@ At the end of the build phase, the lifecycle will forcefully delete everything n ## Rebasing an App -Before a launch image is rebased, the platform must re-run the any stackpacks that were used to build the launch image against the new run-image. It will determine which stackpacks to run using a provided [`stack-group.toml`](https://github.com/buildpacks/spec/blob/main/platform.md#grouptoml-toml). The Build Plan will be stored in a LABEL of the launch-image, which will be serialized back to a `plan.toml` during rebase. To each stackpack, it will pass the build plan derived from the provided [`plan.toml`](https://github.com/buildpacks/spec/blob/main/platform.md#plantoml-toml). +Before a app image is rebased, the platform must re-run the any stackpacks that were used to build the app image against the new run-image. It will determine which stackpacks to run using a provided [`stack-group.toml`](https://github.com/buildpacks/spec/blob/main/platform.md#grouptoml-toml). The Build Plan will be stored in a LABEL of the app image, which will be serialized back to a `plan.toml` during rebase. To each stackpack, it will pass the build plan derived from the provided [`plan.toml`](https://github.com/buildpacks/spec/blob/main/platform.md#plantoml-toml). -The image containing the stack buildpacks and the builder binary must be provided to the rebaser operation as an argument. A platform may choose to provide the same stack buildpacks and builder binary used during the build that create the launch-image being rebased, or it may provide updates versions (which may increase the risk of something failing in the rebase process). +The image containing the stack buildpacks and the builder binary must be provided to the rebaser operation as an argument. A platform may choose to provide the same stack buildpacks and builder binary used during the build that create the app image being rebased, or it may provide updates versions (which may increase the risk of something failing in the rebase process). A platform may choose to store the stack buildpacks and extender binary in any of the following images: -* A companion image of the launch-image +* A companion image of the app image * A builder image that is available when rebasing -* The launch-image itself (store the stack buildpacks and builder binary alongside the application itself) +* The app image itself (store the stack buildpacks and builder binary alongside the application itself) Then, the rebase operation can be performed as normal, while including the stackpack layers as part of the stack. This will be made possible by including the stackpack in the run-image, but because the stackpack detect phase is not run, the operation does not need access to the application source. ## Extending the Run Image -We will introduce a new phase to the lifecycle to support extending the run-image with stack buildpacks. The _extend_ phase will be responsible for running stack buildpacks on the run-image, and creating layers that will be applied to the launch-image. +We will introduce a new phase to the lifecycle to support extending the run-image with stack buildpacks. The _extend_ phase will be responsible for running stack buildpacks on the run-image, and creating layers that will be applied to the app image. ## Example: Apt Buildpack @@ -392,7 +403,7 @@ cache = false Where: -* `paths` = a list of paths to exclude from the launch image layer in the extend phase. During the build phase, these paths will be removed from the filesystem before executing any userspace buildpacks. +* `paths` = a list of paths to exclude from the app image layer in the extend phase. During the build phase, these paths will be removed from the filesystem before executing any userspace buildpacks. * `cache` = if true, the paths will be cached even if they are removed from the layer. 1. Paths not referenced by an `[[excludes]]` entry will be included in the cache _and_ run image (default). From 23a18575d7367eca8a57ea56cb53c89038620bd8 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Mon, 2 Nov 2020 12:53:09 -0600 Subject: [PATCH 110/140] Update 0000-stack-buildpacks.md Signed-off-by: Joe Kutner --- text/0000-stack-buildpacks.md | 24 +++++++++++------------- 1 file changed, 11 insertions(+), 13 deletions(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index 60f5c64b1..c9dbadb31 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -132,16 +132,16 @@ If any required mixins from the Build Plan (any `[[requires]]` tables with `mixi If a stack buildpack provides a mixin that is not required, the stack buildpack MAY pass detection. For each phase, if a stackpack: * provides mixins, and at least one of those mixins are required; it MAY pass * provides mixins, and none of those mixins are required; it MUST be skipped -* provides mixins, and none of those mixins are required, but it also provides another dependency (non-mixin), which is required; it MAY pass +* provides mixins, and none of those mixins are required, but it also provides another dependency (non-mixin), which is required; it MAY pass following the normal build plan rules (TODO add example) * does not provide mixins; it MAY pass -This is in contrast to a userspace buildpack providing a dependency that is not required, which fails detection. If a mixin is required for a [single stage only](https://github.com/buildpacks/rfcs/pull/109) with the `build:` or `run:` prefix, a stack buildpack may provide it for both stages without failing detection. However, it will not be included in the Buildpack Build Plan during the stage where it is not required. +If a mixin is required for a [single stage only](https://github.com/buildpacks/rfcs/pull/109) with the `build:` or `run:` prefix, a stack buildpack may declare that it provides it for both stages without failing detection. However, it will not be included in the Buildpack Build Plan during the stage where it is not required. -During the build phase, the lifecycle will create a build plan containing only the entries required during that stage (build or run). -* If a mixin is required for "run" stage only, it will not appear in the build plan entries during build -* If a mixin is required for "build" stage only, it will not appear in the build plan entries during extend +During the detect phase, the lifecycle will create a build plan containing only the entries required during that stage (build or run) without the [stage-specifier prefix](https://github.com/buildpacks/rfcs/pull/109). +* If a mixin is required for "run" stage only, it will not appear in the buildpack plan entries during build +* If a mixin is required for "build" stage only, it will not appear in the buildpack plan entries during extend -The entries of the build plan during the build and extend phases will have a new `mixin` key. For example: +The entries of the buildpack plan during the build and extend phases will have a new `mixin` key. For example: ``` [[entries]] @@ -152,19 +152,17 @@ A stack buildpack that needs to install mixins must select them from the build p ## Caching and Restoring -During the export phase, the lifecycle will store any snapshot layers created during the build phase in the cache. +During the export phase, the lifecycle will store any excluded cached stackpack layers created during the build or extend phase in the cache. -During the restore phase of the next build, the lifecycle will download the snapshot layer with the cache and store it as a tarball in the `` directory and the `` directory (i.e. it will not extract it). The `restorer` cannot extract the snapshot because it will not run with root privileges. In addition, the `restorer` may run in a different container than the build, which means changes made to the base image are not guaranteed to carry forward. The `builder` will run with root privileges and untar the snapshot and cache, but will drop privileges after executing stackpacks and before running userspace buildpacks. The `extender` will also run with root privileges and untar the cache. +During the restore phase of the next build, the lifecycle will download the excluded cached stackpack layers with the cache and store them as tarballs in the `` directory and the `` directory (i.e. they will not be extracted). The `restorer` cannot extract the these layers because it will not run with root privileges. In addition, the `restorer` may run in a different container than the build, which means changes made to the base image are not guaranteed to carry forward. The `builder` will run with root privileges and untar these layers and cache, but will drop privileges after executing stackpacks and before running userspace buildpacks. The `extender` will also run with root privileges and untar the cache. -During the build and extend phases, the lifecycle will extract and apply snapshot tarballs before running stack buildpacks, but after a snapshot-baseline has been captured. This ensures that all changes from the previous snapshot are preserved even if the stack buildpack does not make any additional changes. - -At the end of the build phase, the lifecycle will forcefully delete everything not in `[[excludes]]` after ALL stackpacks run, before buildpacks run (which implies that it does not need to do this for extend phase). +During the build phase, after the stackpacks have run but before the userspace buildpacks have run, the lifecycle will forcefully delete everything in `[[excludes]]` after ALL stackpacks run, before buildpacks run (which implies that it does not need to do this for extend phase). ## Rebasing an App -Before a app image is rebased, the platform must re-run the any stackpacks that were used to build the app image against the new run-image. It will determine which stackpacks to run using a provided [`stack-group.toml`](https://github.com/buildpacks/spec/blob/main/platform.md#grouptoml-toml). The Build Plan will be stored in a LABEL of the app image, which will be serialized back to a `plan.toml` during rebase. To each stackpack, it will pass the build plan derived from the provided [`plan.toml`](https://github.com/buildpacks/spec/blob/main/platform.md#plantoml-toml). +Before an app image is rebased, the platform must re-run any stackpacks that were used to build the app image, against the new run-image. It will determine which stackpacks to run using a provided [`stack-group.toml`](https://github.com/buildpacks/spec/blob/main/platform.md#grouptoml-toml). The Buildpack Plan will be stored in a LABEL of the app image, which will be serialized back to a `plan.toml` during rebase. To each stackpack, it will pass the buildpack plan derived from the provided [`plan.toml`](https://github.com/buildpacks/spec/blob/main/platform.md#plantoml-toml). -The image containing the stack buildpacks and the builder binary must be provided to the rebaser operation as an argument. A platform may choose to provide the same stack buildpacks and builder binary used during the build that create the app image being rebased, or it may provide updates versions (which may increase the risk of something failing in the rebase process). +The image containing the stack buildpacks and the lifecycle binary must be provided to the rebaser operation as an argument. A platform may choose to provide the same stack buildpacks and lifecycle binary used during the build that create the app image being rebased, or it may provide updated versions (which may increase the risk of something failing in the rebase process). A platform may choose to store the stack buildpacks and extender binary in any of the following images: * A companion image of the app image From eb0d522812acfdf8a12ca94dcbf329d96286cecc Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Thu, 5 Nov 2020 08:21:01 -0600 Subject: [PATCH 111/140] Update 0000-stack-buildpacks.md Signed-off-by: Joe Kutner --- text/0000-stack-buildpacks.md | 24 +++++++++++------------- 1 file changed, 11 insertions(+), 13 deletions(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index c9dbadb31..0eba245d6 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -190,7 +190,7 @@ privileged = true id = "io.buildpacks.stacks.bionic" [stacks.provides] -any = true +mixins = [ "*" ] ``` Its `bin/detect` would have the following contents: @@ -327,8 +327,7 @@ A buildpack may that requires the `jq` package may have it provided by either a ```toml [[or.requires]] -name = "jq" -mixin = true +mixin = "jq" [[or.requires]] name = "jq" @@ -340,6 +339,7 @@ In the future, we plan to enhance the stack buildpack interface with the followi * A `CNB_STACK_TYPE` env var that a stack buildpack can use to behave differently on each part of the stack * Support for `creator`. Because of the new extend phase, it is not possible to easily run the entire buildpack process with a single binary. +* Snapshot caching during the build phase, which would allow stackpack authors cache all changes to the filesystem. # Drawbacks [drawbacks]: #drawbacks @@ -361,11 +361,8 @@ In the future, we plan to enhance the stack buildpack interface with the followi # Unresolved Questions [unresolved-questions]: #unresolved-questions -- what happens if a mixin that already exist on the image is required, how does the lifecycle know? - what about the bill of materials? -- what metadata do we need to do a rebase; what are the schema of the labels? - how do we prevent problems on rebase if the mixins provided by the stack change? -- while the rebase process may get its stackpacks from any image passed to it, do we need to support official ways of producing well-formed run-builders? # Spec. Changes (OPTIONAL) [spec-changes]: #spec-changes @@ -376,9 +373,10 @@ A number of changes to the Platform Specification will be required to execute St Stack buildpacks are identical to other buildpacks, with the following exceptions: -1. The `` directory WILL NOT be used to create layers. +1. The `` directory WILL NOT be used to create arbitrary layers. 1. The working directory WILL NOT contain application source code during the build phase. -1. All changes made to the filesystem during the execution of the stackpack's `bin/build` will be snapshotted and stored as a single layer, with the exception of the following directories: +1. The stack buildpack's `bin/build` will execute on the run-image during the extend phase. +1. All changes made to the filesystem during the execution of the stack buildpack's `bin/build` in the extend phase will be snapshotted and stored as a single layer, with the exception of the following directories: * `/tmp` * `/cnb` @@ -402,15 +400,15 @@ cache = false Where: * `paths` = a list of paths to exclude from the app image layer in the extend phase. During the build phase, these paths will be removed from the filesystem before executing any userspace buildpacks. -* `cache` = if true, the paths will be cached even if they are removed from the layer. +* `cache` = if true, the paths will be cached even if they are removed from the filesystem. -1. Paths not referenced by an `[[excludes]]` entry will be included in the cache _and_ run image (default). -1. Any paths with an `[[excludes]]` entry and `cache = true` will be included in the cache image, but not the run image. -1. Any paths with an `[[excludes]]` entry and `cache = false` will not be included in the cache image or the run image. +1. Paths not referenced by an `[[excludes]]` entry will be included in the app-image layer (default). +1. Any paths with an `[[excludes]]` entry and `cache = true` will be included in the cache image, but not the app image. +1. Any paths with an `[[excludes]]` entry and `cache = false` will not be included in the cache image or the app image. ## buildpack.toml (TOML) - This proposal adds new keys to the `[buildpack]` table in `buildpack.toml`, and a new `mixins` array: +This proposal adds new keys to the `[buildpack]` table in `buildpack.toml`, and a new `mixins` array: ``` [buildpack] From fc21555ec3a70e827b26c7f08a7bc7dc6cc36ab6 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Thu, 5 Nov 2020 13:19:59 -0600 Subject: [PATCH 112/140] Update 0000-stack-buildpacks.md Signed-off-by: Joe Kutner --- text/0000-stack-buildpacks.md | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index 0eba245d6..36bac2113 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -68,6 +68,7 @@ The stackpack interface is similar to the buildpack interface: * The positional arguments for `bin/detect` and `bin/build` are the same as with userspace buildpacks * The environment variables and inputs for `bin/detect` and `bin/build` are the same as with userspace buildpacks (though the values may be different) * The working directory is `/` instead of the app source directory +* The stackpacks will run in the existing build phase as well as a new extend phase. The _extend_ phase will be responsible for running stack buildpacks on the run-image, and creating layers that will be applied to the app image. For each stackpack, the lifecycle will use [snapshotting](https://github.com/GoogleContainerTools/kaniko/blob/master/docs/designdoc.md#snapshotting-snapshotting) to capture changes made during the stackpack's build or extend phases excluding a few specific directories and files. @@ -171,10 +172,6 @@ A platform may choose to store the stack buildpacks and extender binary in any o Then, the rebase operation can be performed as normal, while including the stackpack layers as part of the stack. This will be made possible by including the stackpack in the run-image, but because the stackpack detect phase is not run, the operation does not need access to the application source. -## Extending the Run Image - -We will introduce a new phase to the lifecycle to support extending the run-image with stack buildpacks. The _extend_ phase will be responsible for running stack buildpacks on the run-image, and creating layers that will be applied to the app image. - ## Example: Apt Buildpack (**note**: this is only an example. A real Apt Buildpack would probably be more robust). From 414f5352a86a37bdceca64dc50e591d8b023d917 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Thu, 5 Nov 2020 14:08:19 -0600 Subject: [PATCH 113/140] Update 0000-stack-buildpacks.md Signed-off-by: Joe Kutner --- text/0000-stack-buildpacks.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index 36bac2113..6cde96a2b 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -172,6 +172,10 @@ A platform may choose to store the stack buildpacks and extender binary in any o Then, the rebase operation can be performed as normal, while including the stackpack layers as part of the stack. This will be made possible by including the stackpack in the run-image, but because the stackpack detect phase is not run, the operation does not need access to the application source. +## Stack Platform Dir + +TODO + ## Example: Apt Buildpack (**note**: this is only an example. A real Apt Buildpack would probably be more robust). @@ -360,6 +364,7 @@ In the future, we plan to enhance the stack buildpack interface with the followi - what about the bill of materials? - how do we prevent problems on rebase if the mixins provided by the stack change? +- how does detect know the run-image mixins? # Spec. Changes (OPTIONAL) [spec-changes]: #spec-changes From f710fac8b39eabdb6f135bad682f6e2ae2e75fdf Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Wed, 11 Nov 2020 15:05:01 -0600 Subject: [PATCH 114/140] Update 0000-stack-buildpacks.md Signed-off-by: Joe Kutner --- text/0000-stack-buildpacks.md | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index 6cde96a2b..6ff1ecbb9 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -176,6 +176,12 @@ Then, the rebase operation can be performed as normal, while including the stack TODO +seperate env vars for the stack (--stack-env) + +no accidental envars coming from other buildpacks or other places + + + ## Example: Apt Buildpack (**note**: this is only an example. A real Apt Buildpack would probably be more robust). @@ -340,7 +346,7 @@ In the future, we plan to enhance the stack buildpack interface with the followi * A `CNB_STACK_TYPE` env var that a stack buildpack can use to behave differently on each part of the stack * Support for `creator`. Because of the new extend phase, it is not possible to easily run the entire buildpack process with a single binary. -* Snapshot caching during the build phase, which would allow stackpack authors cache all changes to the filesystem. +* Snapshot caching during the build and extend phase, which would allow stackpack authors cache all changes to the filesystem. # Drawbacks [drawbacks]: #drawbacks @@ -363,6 +369,8 @@ In the future, we plan to enhance the stack buildpack interface with the followi [unresolved-questions]: #unresolved-questions - what about the bill of materials? + - extend phase artifacts go in the BOM + - build phase go in report.toml - how do we prevent problems on rebase if the mixins provided by the stack change? - how does detect know the run-image mixins? From c882de0fcef036d68704011ffedc8568ba0450c0 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Thu, 12 Nov 2020 09:23:21 -0600 Subject: [PATCH 115/140] Update 0000-stack-buildpacks.md Added stack-specific platform directory Signed-off-by: Joe Kutner --- text/0000-stack-buildpacks.md | 29 ++++++++++++++++------------- 1 file changed, 16 insertions(+), 13 deletions(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index 6ff1ecbb9..d31a6a815 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -133,9 +133,11 @@ If any required mixins from the Build Plan (any `[[requires]]` tables with `mixi If a stack buildpack provides a mixin that is not required, the stack buildpack MAY pass detection. For each phase, if a stackpack: * provides mixins, and at least one of those mixins are required; it MAY pass * provides mixins, and none of those mixins are required; it MUST be skipped -* provides mixins, and none of those mixins are required, but it also provides another dependency (non-mixin), which is required; it MAY pass following the normal build plan rules (TODO add example) +* provides mixins, and none of those mixins are required, but it also provides another dependency (non-mixin), which is required; it MAY pass following the normal build plan rules * does not provide mixins; it MAY pass +As an example of these resolution rules, consider a stackpack that provides a mixin and a dependency. TODO + If a mixin is required for a [single stage only](https://github.com/buildpacks/rfcs/pull/109) with the `build:` or `run:` prefix, a stack buildpack may declare that it provides it for both stages without failing detection. However, it will not be included in the Buildpack Build Plan during the stage where it is not required. During the detect phase, the lifecycle will create a build plan containing only the entries required during that stage (build or run) without the [stage-specifier prefix](https://github.com/buildpacks/rfcs/pull/109). @@ -172,15 +174,11 @@ A platform may choose to store the stack buildpacks and extender binary in any o Then, the rebase operation can be performed as normal, while including the stackpack layers as part of the stack. This will be made possible by including the stackpack in the run-image, but because the stackpack detect phase is not run, the operation does not need access to the application source. -## Stack Platform Dir - -TODO - -seperate env vars for the stack (--stack-env) - -no accidental envars coming from other buildpacks or other places +## Stack Platform Directory +The `detect` and `build` phases will be provided with a stack-specific platform directory, instead of the userspace platform directory. In this way, the lifecycle can ensure that no environment variables or other platform-specific extensions are coming from other buildpacks or other sources that may impact the stackpacks ability to successful rebase in the future. +The new stack-specific platform directory will require new options for the lifecycle detect and build interfaces, but the stackpacks `detect` and `build` executables will retain the same interface (i.e. they simply get a different platform dir than other buildpacks). ## Example: Apt Buildpack @@ -490,7 +488,8 @@ Usage: ... [-stack-buildpacks ] \ [-stack-group ] \ - [-stack-order ] + [-stack-order ] \ + [-stack-platform ] ``` ##### Inputs @@ -500,6 +499,7 @@ Usage: | `` | `CNB_STACK_BUILDPACKS_DIR` | `/cnb/stack/buildpacks` | Path to stack buildpacks directory (see [Buildpacks Directory Layout] | `` | `CNB_STACK_GROUP_PATH` | `./stack-group.toml` | Path to output group definition(#buildpacks-directory-layout)) | ``. | `CNB_STACK_ORDER_PATH`. | `/cnb/stack/order.toml`| Path to order definition (see order.toml) +| `` | `CNB_STACK_PLATFORM_DIR` | `/stack-platform` | Path to stack-specific platform directory ##### Outputs | Output | Description @@ -516,7 +516,8 @@ Usage: /cnb/lifecycle/builder \ ... [-stack-buildpacks ] \ - [-stack-group ] + [-stack-group ] \ + [-stack-platform ] ``` ##### Inputs @@ -525,6 +526,7 @@ Usage: | ... | | | | `` | `CNB_STACK_BUILDPACKS_DIR` | `/cnb/stack/buildpacks` | Path to stack buildpacks directory (see [Buildpacks Directory Layout] | `` | `CNB_STACK_GROUP_PATH` | `./stack-group.toml` | Path to output group definition(#buildpacks-directory-layout)) +| `` | `CNB_STACK_PLATFORM_DIR` | `/stack-platform` | Path to stack-specific platform directory ##### Outputs | Output | Description @@ -550,6 +552,7 @@ Usage: | `` | `CNB_STACK_LAYERS_DIR` | `/stack-layers` | Path to stack layers directory from extend phase + #### `rebaser` Usage: ``` @@ -565,7 +568,7 @@ Usage: | ... | | | | `` | `CNB_CACHE_IMAGE` | | Reference to a cache image in an OCI image registry | `` | `CNB_PLAN_PATH` | `./plan.toml` | Path to resolved build plan (see [`plan.toml`](#plantoml-toml)) -| `` | `CNB_PLATFORM_DIR` | `/platform` | Path to platform directory +| `` | `CNB_STACK_PLATFORM_DIR` | `/stack-platform` | Path to stack-specific platform directory | `` | `CNB_STACK_GROUP_PATH` | `./stack-group.toml` | Path to group definition(#buildpacks-directory-layout)) of buildpacks that run during the extend phase that created the image being rebased. #### `extender` @@ -577,7 +580,7 @@ Usage: [-layers ] \ [-log-level ] \ [-plan ] \ - [-platform ] + [-stack-platform ] ``` ##### Inputs @@ -586,7 +589,7 @@ Usage: | `` | `CNB_LAYERS_DIR` | `/layers` | Path to layers directory | `` | `CNB_LOG_LEVEL` | `info` | Log Level | `` | `CNB_PLAN_PATH` | `./plan.toml` | Path to resolved build plan (see [`plan.toml`](#plantoml-toml)) -| `` | `CNB_PLATFORM_DIR` | `/platform` | Path to platform directory +| `` | `CNB_STACK_PLATFORM_DIR` | `/stack-platform` | Path to stack-specific platform directory | `` | `CNB_STACK_BUILDPACKS_DIR` | `/cnb/stack/buildpacks` | Path to stack buildpacks directory (see [Buildpacks Directory Layout] | `` | `CNB_STACK_GROUP_PATH` | `./stack-group.toml` | Path to output group definition(#buildpacks-directory-layout)) From b6c646ea27ef9404aeb7fc880df0e450d9ce51d7 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Thu, 12 Nov 2020 10:26:42 -0600 Subject: [PATCH 116/140] Update 0000-stack-buildpacks.md Added examples of mixin and build plan resolution. Signed-off-by: Joe Kutner --- text/0000-stack-buildpacks.md | 56 +++++++++++++++++++++++++++++++++-- 1 file changed, 53 insertions(+), 3 deletions(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index d31a6a815..e37dec111 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -42,7 +42,7 @@ A stack provider may choose to include stack buildpacks with the stack they dist 1. The lifecycle will merge and run the detect phase for all stackpacks defined in the `/cnb/stack/order.toml` and for all userspace buildpacks defined in `/cnb/order.toml`. * During the build phase (potentially in parallel to extend phase): - 1. The lifecycle will execute the stack buildpack build phase for all passing stackpack(s) as root. + 1. The lifecycle will execute the stack buildpack build phase with root privileges for all passing stackpack(s). 1. The lifecycle will drop privilidges and continue the build phase as normal (running userspace buildpacks). * During the extend phase (potentially in parallel to build phase): @@ -136,8 +136,6 @@ If a stack buildpack provides a mixin that is not required, the stack buildpack * provides mixins, and none of those mixins are required, but it also provides another dependency (non-mixin), which is required; it MAY pass following the normal build plan rules * does not provide mixins; it MAY pass -As an example of these resolution rules, consider a stackpack that provides a mixin and a dependency. TODO - If a mixin is required for a [single stage only](https://github.com/buildpacks/rfcs/pull/109) with the `build:` or `run:` prefix, a stack buildpack may declare that it provides it for both stages without failing detection. However, it will not be included in the Buildpack Build Plan during the stage where it is not required. During the detect phase, the lifecycle will create a build plan containing only the entries required during that stage (build or run) without the [stage-specifier prefix](https://github.com/buildpacks/rfcs/pull/109). @@ -153,6 +151,58 @@ mixin = "libpq" A stack buildpack that needs to install mixins must select them from the build plan. +### Example: PostgreSQL Certs + +As an example of these resolution rules, consider an Apt stackpack that provide any (`*`) mixin. During the detect phase, the stackpack will pass in the following cases: +* a mixin is required by a buildpack +* a mixin is [required by an app](https://github.com/buildpacks/rfcs/pull/112) + +The example stackpack will be skipped after detection in the following cases: +* no mixins are required by any component of the build +* A mixin is required by a buildpack, but the stack already provides that mixin. +* A mixin is required by an app, but the stack already provides that mixin. + +### Example: PostgreSQL Certs + +As another example of these resolution rules, consider a stackpack that provides the `libpq` mixin and a build plan dependency called `db-cert`. During the detect phase, the stackpack will pass in the following cases: +* a Ruby app requires the `libpq` mixin and the `db-cert` +* a Java app requires only the `db-cert` +* a Python app requires the `db-cert` and the stack already provides the `libpq` mixin + +The example stackpack will be skipped after detection in the following cases: +* an app requires the `libpq` mixin but not the `db-cert` +* an app requires neither the `libpq` mixin nor the `db-cert` + +### Example: Chrome and Chromedriver + +As another example of these resolution rules, consider a stackpack that provides the `google-chrome-stable` mixin and a build plan dependency called `chromedriver`. During the detect phase, the stackpack will pass detection in the following cases: +* an app requires the `google-chrome-stable` mixin and the `chromedriver` +* an app requires only the `chromedriver` +* an app requires the `chromedriver` and the stack already provides the `google-chrome-stable` mixin + +The example stackpack will be skipped after detection in the following cases: +* an app requires the `google-chrome-stable` mixin but not the `chromedriver` +* an app requires neither the `google-chrome-stable` mixin nor the `chromedriver` + +During the build phase the example stackpack will be given a [buildpack plan](https://github.com/buildpacks/spec/blob/main/buildpack.md#buildpack-plan-toml) containing the entries it must provide. For example, if both dependencies are required it will receive: + +```toml +[[entries]] +mixin = "google-chrome-stable" + +[[entries]] +name = "chromedriver" +``` + +If only the `chromedriver` is required, or the `google-chrome-stable` is already provided by the stack, the build phase would receive: + +```toml +[[entries]] +name = "chromedriver" +``` + +This implies that a build may fail or an app may not work at runtime if that app requires only the `chromedriver` and the `google-chrome-stable` is not provided by the stack. + ## Caching and Restoring During the export phase, the lifecycle will store any excluded cached stackpack layers created during the build or extend phase in the cache. From fcf6d0f7219c2a84efdae7a50ac3ee7c4169cd17 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Thu, 12 Nov 2020 10:27:28 -0600 Subject: [PATCH 117/140] Update 0000-stack-buildpacks.md Signed-off-by: Joe Kutner --- text/0000-stack-buildpacks.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index e37dec111..cd064d8d1 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -151,7 +151,7 @@ mixin = "libpq" A stack buildpack that needs to install mixins must select them from the build plan. -### Example: PostgreSQL Certs +### Example: Any Mixin Buildpack As an example of these resolution rules, consider an Apt stackpack that provide any (`*`) mixin. During the detect phase, the stackpack will pass in the following cases: * a mixin is required by a buildpack From c6d6c8a5ae874be8e01f00bde71a0c15bba317f2 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Thu, 12 Nov 2020 14:03:29 -0600 Subject: [PATCH 118/140] Update 0000-stack-buildpacks.md Signed-off-by: Joe Kutner --- text/0000-stack-buildpacks.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index cd064d8d1..cf2592481 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -201,6 +201,8 @@ If only the `chromedriver` is required, or the `google-chrome-stable` is already name = "chromedriver" ``` +If the stack does not already provide `google-chrome-stable`, the stackpack will install it anyways.... TODO + This implies that a build may fail or an app may not work at runtime if that app requires only the `chromedriver` and the `google-chrome-stable` is not provided by the stack. ## Caching and Restoring @@ -421,6 +423,7 @@ In the future, we plan to enhance the stack buildpack interface with the followi - build phase go in report.toml - how do we prevent problems on rebase if the mixins provided by the stack change? - how does detect know the run-image mixins? +- If a mixin is required for build, but is not needed for extend, does the stackpack execute in both phases (and visa versa)? # Spec. Changes (OPTIONAL) [spec-changes]: #spec-changes From 44218db3b3042fb8bacb1a9ac384f4872aca165a Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Thu, 12 Nov 2020 14:04:36 -0600 Subject: [PATCH 119/140] Update 0000-stack-buildpacks.md Signed-off-by: Joe Kutner --- text/0000-stack-buildpacks.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index cf2592481..3475716a9 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -201,9 +201,7 @@ If only the `chromedriver` is required, or the `google-chrome-stable` is already name = "chromedriver" ``` -If the stack does not already provide `google-chrome-stable`, the stackpack will install it anyways.... TODO - -This implies that a build may fail or an app may not work at runtime if that app requires only the `chromedriver` and the `google-chrome-stable` is not provided by the stack. +If the stack does not already provide `google-chrome-stable`, the stackpack may install it anyways. This prevents the case where an app may not work at runtime if that app requires only the `chromedriver` and the `google-chrome-stable` is not provided by the stack. ## Caching and Restoring From 91c396ac1bc55c742728ae3fd5a3bd7f3df9105b Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Thu, 10 Dec 2020 15:47:27 -0600 Subject: [PATCH 120/140] Rename userspace buildpacks to application buildpacks Signed-off-by: Joe Kutner --- text/0000-stack-buildpacks.md | 34 +++++++++++++++++----------------- 1 file changed, 17 insertions(+), 17 deletions(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index 3475716a9..862589e97 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -24,11 +24,11 @@ However, many applications and buildpacks require modifications to the stack the [what-it-is]: #what-it-is - *stack buildpack* - (a.k.a. stackpack) a type of buildpack that runs with root privileges against the stack image(s) instead of an app. Stack buildpacks must not make changes to the build and run images that either violate stack [compatibility guarantees](https://github.com/buildpacks/spec/blob/main/platform.md#compatibility-guarantees) or violate the contract defined by that stack's author. -- *userspace buildpack* - the traditional definition of a buildpack (i.e. does not run as root, and runs against an app) +- *application buildpack* - the traditional definition of a buildpack (i.e. does not run as root, and runs against an app) - *app image* - the final image produced by a buildpack build. It consists of a run-image combined with layers created by buildpacks. - *extend phase* - a new buildpack phase in which a stack buildpack may create a layer for the app image. -A new type of buildpack, called a stack buildpack, may run against a stack (both build and run images) in order to extend it in ways that are only possible by running privileged commands. Unlike userspace buildpacks, stack buildpack can modify any path (with exceptions) on the filesystem. Userspace buildpack can only create/modify disjoint layers (either by adding a dir to `` or modifying an app slice), which makes possible features like individual layer reuse that is independent or ordering. +A new type of buildpack, called a stack buildpack, may run against a stack (both build and run images) in order to extend it in ways that are only possible by running privileged commands. Unlike application buildpacks, stack buildpack can modify any path (with exceptions) on the filesystem. Application buildpacks can only create/modify disjoint layers (either by adding a dir to `` or modifying an app slice), which makes possible features like individual layer reuse that is independent or ordering. A stackpack may also define a list of mixins that it can provide to the stack, or indicate that it can provide _any_ mixin. In this way, a stack that is missing a mixin required by a buildpack may have that mixin provided by a stack buildpack. @@ -39,11 +39,11 @@ A stack provider may choose to include stack buildpacks with the stack they dist * During the detect phase: 1. The lifecycle will compare the list of required mixins to the list of mixins provided by stack and stack buildpacks in accordance with [stage-specific mixin rules](https://github.com/buildpacks/rfcs/pull/109). If any mixins are still not provided, the build will fail. To accomplish this, the list of run-time and build-time mixins that are already present in the run image and build image must be provided to the detector. - 1. The lifecycle will merge and run the detect phase for all stackpacks defined in the `/cnb/stack/order.toml` and for all userspace buildpacks defined in `/cnb/order.toml`. + 1. The lifecycle will merge and run the detect phase for all stackpacks defined in the `/cnb/stack/order.toml` and for all application buildpacks defined in `/cnb/order.toml`. * During the build phase (potentially in parallel to extend phase): 1. The lifecycle will execute the stack buildpack build phase with root privileges for all passing stackpack(s). - 1. The lifecycle will drop privilidges and continue the build phase as normal (running userspace buildpacks). + 1. The lifecycle will drop privilidges and continue the build phase as normal (running application buildpacks). * During the extend phase (potentially in parallel to build phase): 1. During the extend phase, stack buildpacks that passed detection will run against the run images accordingly (see details below). @@ -58,21 +58,21 @@ A stack provider may choose to include stack buildpacks with the stack they dist * Can only create one layer that is exported in the app image * May include, in the created layer, modifications to any part of the filesystem, excluding ``, ``, ``, and `` directories as well as other directories [listed below](#spec-changes-optional). * Must not access the application source code during the `build` phase -* Must run before all userspace buildpacks if it passes detection +* Must run before all application buildpacks if it passes detection * Must run against both the build and run images if it passes detection for that stack type * Must be distributed with the stack build image, and may be distributed with the stack run image * May not create layers using the `` directory The stackpack interface is similar to the buildpack interface: * The same `bin/detect` and `bin/build` scripts are required -* The positional arguments for `bin/detect` and `bin/build` are the same as with userspace buildpacks -* The environment variables and inputs for `bin/detect` and `bin/build` are the same as with userspace buildpacks (though the values may be different) +* The positional arguments for `bin/detect` and `bin/build` are the same as with application buildpacks +* The environment variables and inputs for `bin/detect` and `bin/build` are the same as with application buildpacks (though the values may be different) * The working directory is `/` instead of the app source directory * The stackpacks will run in the existing build phase as well as a new extend phase. The _extend_ phase will be responsible for running stack buildpacks on the run-image, and creating layers that will be applied to the app image. For each stackpack, the lifecycle will use [snapshotting](https://github.com/GoogleContainerTools/kaniko/blob/master/docs/designdoc.md#snapshotting-snapshotting) to capture changes made during the stackpack's build or extend phases excluding a few specific directories and files. -All of the captured changes in the extend phase will be included in a single layer produced as output from the stackpack, which will be mounted into the `/run-layers` directory of the export container. Any changes performed by the stack buildpack to the build image will persist through execution of userspace buildpacks, but not exported as a layer. +All of the captured changes in the extend phase will be included in a single layer produced as output from the stackpack, which will be mounted into the `/run-layers` directory of the export container. Any changes performed by the stack buildpack to the build image will persist through execution of application buildpacks, but not exported as a layer. The `` dir MAY NOT be used to create arbitrary layers. @@ -84,11 +84,11 @@ A stack can provide stackpacks by including them in the `/cnb/stack/buildpacks` id = "" ``` -A stackpack will only execute if it passes detection. When the stackpack is executed, its detect and build scripts use the same parameters as the userspace buildpacks. +A stackpack will only execute if it passes detection. When the stackpack is executed, its detect and build scripts use the same parameters as the application buildpacks. The stackpack's snapshot layer may be enriched by writing a `stack-layer.toml` file. The `stack-layer.toml` may define globs of files to be excluded from the image when it is _exported_. Any excluded path may also be marked as _cached_, so that those excluded paths are recovered before the build or extend phase. The term _exported_ is defined as: -* *Exported for build-time build*: A given path is excluded at userspace buildpack build-time, and recovered the next time the build image is extended with the stackpack. +* *Exported for build-time build*: A given path is excluded at application buildpack build-time, and recovered the next time the build image is extended with the stackpack. * *Exported for build-time run*: A given path is excluded from the final image, and restored the next time the run image is extended with the stackpack (either rebase or rebuild). * *Exported for rebase run*: A given path is excluded from the rebased image, and recovered the next time the run image is extended with the stackpack (either rebase or rebuild). @@ -112,14 +112,14 @@ mixins = [ "" ] A stack buildpack MAY NOT require any entries in the build plan (neither mixins nor non-mixins). This ensures that we do not need to re-run the detect phase. -A userspace buildpack MAY require mixins in the build plan +An application buildpack MAY require mixins in the build plan ``` [[requires]] mixin = "" ``` -A userspace buildpack MAY NOT provide mixins in the build plan. +An application buildpack MAY NOT provide mixins in the build plan. ### Resolving Mixins @@ -207,9 +207,9 @@ If the stack does not already provide `google-chrome-stable`, the stackpack may During the export phase, the lifecycle will store any excluded cached stackpack layers created during the build or extend phase in the cache. -During the restore phase of the next build, the lifecycle will download the excluded cached stackpack layers with the cache and store them as tarballs in the `` directory and the `` directory (i.e. they will not be extracted). The `restorer` cannot extract the these layers because it will not run with root privileges. In addition, the `restorer` may run in a different container than the build, which means changes made to the base image are not guaranteed to carry forward. The `builder` will run with root privileges and untar these layers and cache, but will drop privileges after executing stackpacks and before running userspace buildpacks. The `extender` will also run with root privileges and untar the cache. +During the restore phase of the next build, the lifecycle will download the excluded cached stackpack layers with the cache and store them as tarballs in the `` directory and the `` directory (i.e. they will not be extracted). The `restorer` cannot extract the these layers because it will not run with root privileges. In addition, the `restorer` may run in a different container than the build, which means changes made to the base image are not guaranteed to carry forward. The `builder` will run with root privileges and untar these layers and cache, but will drop privileges after executing stackpacks and before running application buildpacks. The `extender` will also run with root privileges and untar the cache. -During the build phase, after the stackpacks have run but before the userspace buildpacks have run, the lifecycle will forcefully delete everything in `[[excludes]]` after ALL stackpacks run, before buildpacks run (which implies that it does not need to do this for extend phase). +During the build phase, after the stackpacks have run but before the application buildpacks have run, the lifecycle will forcefully delete everything in `[[excludes]]` after ALL stackpacks run, before buildpacks run (which implies that it does not need to do this for extend phase). ## Rebasing an App @@ -226,7 +226,7 @@ Then, the rebase operation can be performed as normal, while including the stack ## Stack Platform Directory -The `detect` and `build` phases will be provided with a stack-specific platform directory, instead of the userspace platform directory. In this way, the lifecycle can ensure that no environment variables or other platform-specific extensions are coming from other buildpacks or other sources that may impact the stackpacks ability to successful rebase in the future. +The `detect` and `build` phases will be provided with a stack-specific platform directory, instead of the application platform directory. In this way, the lifecycle can ensure that no environment variables or other platform-specific extensions are coming from other buildpacks or other sources that may impact the stackpacks ability to successful rebase in the future. The new stack-specific platform directory will require new options for the lifecycle detect and build interfaces, but the stackpacks `detect` and `build` executables will retain the same interface (i.e. they simply get a different platform dir than other buildpacks). @@ -378,7 +378,7 @@ exit 0 ## Example: jq mixin -A buildpack may that requires the `jq` package may have it provided by either a stack, stack buildpack, or userspace buildpack. +A buildpack may that requires the `jq` package may have it provided by either a stack, stack buildpack, or application buildpack. ```toml [[or.requires]] @@ -458,7 +458,7 @@ cache = false Where: -* `paths` = a list of paths to exclude from the app image layer in the extend phase. During the build phase, these paths will be removed from the filesystem before executing any userspace buildpacks. +* `paths` = a list of paths to exclude from the app image layer in the extend phase. During the build phase, these paths will be removed from the filesystem before executing any application buildpacks. * `cache` = if true, the paths will be cached even if they are removed from the filesystem. 1. Paths not referenced by an `[[excludes]]` entry will be included in the app-image layer (default). From 009f739d251e27e509f3ef15a655b6d1edc5b88e Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Thu, 17 Dec 2020 13:17:28 -0600 Subject: [PATCH 121/140] Update text/0000-stack-buildpacks.md Signed-off-by: Joe Kutner Co-authored-by: Emily Casey --- text/0000-stack-buildpacks.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index 862589e97..a17bf68f7 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -665,4 +665,4 @@ Usage: - The lifecycle SHALL execute all stack buildpacks in the order defined in `` according to the process outlined in the [Buildpack Interface Specification](buildpack.md). - The lifecycle SHALL add all invoked stack buildpacks to`/config/metadata.toml`. -- The lifecycle SHALL aggregate all `processes` and BOM entries returned by buildpacks in `/config/metadata.toml`. +- The lifecycle SHALL aggregate all BOM entries returned by buildpacks in `/config/metadata.toml`. From 90bdf5f19db1f9665d363bb3fe68803ad0012642 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Thu, 17 Dec 2020 13:17:58 -0600 Subject: [PATCH 122/140] Update text/0000-stack-buildpacks.md Signed-off-by: Joe Kutner Co-authored-by: Emily Casey --- text/0000-stack-buildpacks.md | 1 - 1 file changed, 1 deletion(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index a17bf68f7..d2f234f89 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -583,7 +583,6 @@ Usage: | Output | Description |--------------------------------------------|---------------------------------------------- | ... | -| `/.tgz` | Layer snapshot (see [Buildpack Interface Specfication](buildpack.md) | `/stack-layer.toml` | Layer snaphot metadata #### `exporter` From 5bca374c1e03b0dffba8aa2c979012b9fe9bcb84 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Thu, 17 Dec 2020 13:18:18 -0600 Subject: [PATCH 123/140] Update text/0000-stack-buildpacks.md Signed-off-by: Joe Kutner Co-authored-by: Emily Casey --- text/0000-stack-buildpacks.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index d2f234f89..2cdc743eb 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -28,7 +28,7 @@ However, many applications and buildpacks require modifications to the stack the - *app image* - the final image produced by a buildpack build. It consists of a run-image combined with layers created by buildpacks. - *extend phase* - a new buildpack phase in which a stack buildpack may create a layer for the app image. -A new type of buildpack, called a stack buildpack, may run against a stack (both build and run images) in order to extend it in ways that are only possible by running privileged commands. Unlike application buildpacks, stack buildpack can modify any path (with exceptions) on the filesystem. Application buildpacks can only create/modify disjoint layers (either by adding a dir to `` or modifying an app slice), which makes possible features like individual layer reuse that is independent or ordering. +A new type of buildpack, called a stack buildpack, may run against a stack (both build and run images) in order to extend it in ways that are only possible by running privileged commands. Unlike application buildpacks, stack buildpack can modify any path (with exceptions) on the filesystem. Application buildpacks can only create/modify disjoint layers (either by adding a dir to `` or modifying an app slice), which makes possible features like individual layer reuse that is independent of ordering. A stackpack may also define a list of mixins that it can provide to the stack, or indicate that it can provide _any_ mixin. In this way, a stack that is missing a mixin required by a buildpack may have that mixin provided by a stack buildpack. From 3bd08cf198c2e63b1c73091d55b895770ea6ef9f Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Thu, 17 Dec 2020 13:18:30 -0600 Subject: [PATCH 124/140] Update text/0000-stack-buildpacks.md Signed-off-by: Joe Kutner Co-authored-by: Emily Casey --- text/0000-stack-buildpacks.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index 2cdc743eb..23a24f7c1 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -46,7 +46,7 @@ A stack provider may choose to include stack buildpacks with the stack they dist 1. The lifecycle will drop privilidges and continue the build phase as normal (running application buildpacks). * During the extend phase (potentially in parallel to build phase): - 1. During the extend phase, stack buildpacks that passed detection will run against the run images accordingly (see details below). + 1. During the extend phase, stack buildpacks that passed detection will run against the run image accordingly (see details below). # How it Works [how-it-works]: #how-it-works From 316e3d472626c1804ff1bc2d6bd1ba81233d0483 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Thu, 17 Dec 2020 13:19:48 -0600 Subject: [PATCH 125/140] Update text/0000-stack-buildpacks.md Signed-off-by: Joe Kutner Co-authored-by: Emily Casey --- text/0000-stack-buildpacks.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index 23a24f7c1..378d38bae 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -330,7 +330,7 @@ update-ca-certificates ### CA Cert Provider Buildpack -The stackpack must be used with a buildpack that provides a certificate(s) for it to install. That buildpack would have the following `buildpack.toml`: +The stackpack must be used with a buildpack that requires a certificate(s) for it to install. That buildpack would have the following `buildpack.toml`: ```toml [buildpack] From ef5d286365e7b4d6990533a3a09027443dde3e4c Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Thu, 17 Dec 2020 13:20:48 -0600 Subject: [PATCH 126/140] Update 0000-stack-buildpacks.md Signed-off-by: Joe Kutner --- text/0000-stack-buildpacks.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index 378d38bae..c7048a2c7 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -285,7 +285,7 @@ This would instruct the Apt Buildpack to install the libpq package. ## Example: CA Certificate Buildpack -Support for custom CA Certificates can be accomplished with two buildpacks: a stackpack that can install the cert, and a normal buildpack that can provide a cert in the build plan. +Support for custom CA Certificates can be accomplished with two buildpacks: a stackpack that can install the cert, and an application buildpack that can provide a cert in the build plan. ### CA Cert Installer Stackpack @@ -328,7 +328,7 @@ done update-ca-certificates ``` -### CA Cert Provider Buildpack +### CA Cert Injector Buildpack The stackpack must be used with a buildpack that requires a certificate(s) for it to install. That buildpack would have the following `buildpack.toml`: From cb7a3eb0802c773c0b37a3ae24cbb1e0f25a1012 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Thu, 17 Dec 2020 13:22:24 -0600 Subject: [PATCH 127/140] Update text/0000-stack-buildpacks.md Signed-off-by: Joe Kutner Co-authored-by: Emily Casey --- text/0000-stack-buildpacks.md | 1 - 1 file changed, 1 deletion(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index c7048a2c7..7de2c817f 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -411,7 +411,6 @@ In the future, we plan to enhance the stack buildpack interface with the followi # Prior Art [prior-art]: #prior-art -- The term "non-idempotent" is used in section 9.1.2 of [Hypertext Transfer Protocol -- HTTP/1.1 (RFC 2616)](https://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html) # Unresolved Questions [unresolved-questions]: #unresolved-questions From 6bd8d20bd6e5bd7239b84979d179eacba1f2d8ba Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Fri, 18 Dec 2020 09:41:05 -0600 Subject: [PATCH 128/140] Update 0000-stack-buildpacks.md Signed-off-by: Joe Kutner --- text/0000-stack-buildpacks.md | 1 + 1 file changed, 1 insertion(+) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index 7de2c817f..5db4cfd83 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -411,6 +411,7 @@ In the future, we plan to enhance the stack buildpack interface with the followi # Prior Art [prior-art]: #prior-art +* [Heroku Apt Buildpack](https://github.com/heroku/heroku-buildpack-apt) # Unresolved Questions [unresolved-questions]: #unresolved-questions From 848f3488a111f8722d0b5847358e2e95d0b23d3f Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Fri, 18 Dec 2020 09:46:07 -0600 Subject: [PATCH 129/140] Change extender exit codes to 800 Signed-off-by: Joe Kutner --- text/0000-stack-buildpacks.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index 5db4cfd83..4ff34a687 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -659,8 +659,8 @@ Usage: | `11` | Platform API incompatibility error | `12` | Buildpack API incompatibility error | `1-10`, `13-99` | Generic lifecycle errors -| `401` | Buildpack build error -| `400`, `402-499`| Build-specific lifecycle errors +| `801` | Buildpack build error +| `800`, `802-899`| Build-specific lifecycle errors - The lifecycle SHALL execute all stack buildpacks in the order defined in `` according to the process outlined in the [Buildpack Interface Specification](buildpack.md). - The lifecycle SHALL add all invoked stack buildpacks to`/config/metadata.toml`. From 146d0d5bf6936335fc276ca06e2982ec792eb6e9 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Fri, 18 Dec 2020 09:54:13 -0600 Subject: [PATCH 130/140] Added stack-cache-tar.gz to store excludes paths Signed-off-by: Joe Kutner --- text/0000-stack-buildpacks.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index 4ff34a687..f6bd91937 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -583,7 +583,8 @@ Usage: | Output | Description |--------------------------------------------|---------------------------------------------- | ... | -| `/stack-layer.toml` | Layer snaphot metadata +| `/stack-cache.tar.gz` | Cache snaphot for each stack buildpack +| `/stack-layer.toml` | Layer snaphot metadata for each stack buildpack #### `exporter` Usage: @@ -650,7 +651,8 @@ Usage: | `/dev/stdout` | Logs (info) | `/dev/stderr` | Logs (warnings, errors) | `/.tgz` | Layer snapshot (see [Buildpack Interface Specfication](buildpack.md) -| `/stack-layer.toml` | Layer snaphot metadata +| `/stack-cache.tar.gz` | Cache snaphot for each stack buildpack +| `/stack-layer.toml` | Layer snaphot metadata for each stack buildpack | `/config/metadata.toml` | Build metadata (see [`metadata.toml`](#metadatatoml-toml)) | Exit Code | Result| From 5aa7ff89b27d213a66c3ac655b3448c2a776ec4f Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Fri, 18 Dec 2020 09:55:40 -0600 Subject: [PATCH 131/140] Update text/0000-stack-buildpacks.md Signed-off-by: Joe Kutner Co-authored-by: Emily Casey --- text/0000-stack-buildpacks.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index f6bd91937..9e717bbc1 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -422,6 +422,9 @@ In the future, we plan to enhance the stack buildpack interface with the followi - how do we prevent problems on rebase if the mixins provided by the stack change? - how does detect know the run-image mixins? - If a mixin is required for build, but is not needed for extend, does the stackpack execute in both phases (and visa versa)? +- How is the buildpack plan provided to a stack buildpack during the extend phase of a rebase operation? +- Will `io.buildpacks.stack.mixins` on the app image be updated to include stack buildpack-provided mixins? +- During a rebase, if the new run image statically includes a mixin that was previously provided by a stack buildpack, should the providing stack buildpack rerun? # Spec. Changes (OPTIONAL) [spec-changes]: #spec-changes From a8ecc01ad7f54df7dc0a2ade0b2e87b230805b79 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Fri, 18 Dec 2020 10:00:26 -0600 Subject: [PATCH 132/140] Scope each stack snapshot and metadata to buildpack-id dir Signed-off-by: Joe Kutner --- text/0000-stack-buildpacks.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index 9e717bbc1..79ef4d928 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -586,8 +586,8 @@ Usage: | Output | Description |--------------------------------------------|---------------------------------------------- | ... | -| `/stack-cache.tar.gz` | Cache snaphot for each stack buildpack -| `/stack-layer.toml` | Layer snaphot metadata for each stack buildpack +| `//stack-cache.tgz` | Cache snaphot for each stack buildpack +| `//stack-layer.toml` | Layer snaphot metadata for each stack buildpack #### `exporter` Usage: @@ -653,9 +653,9 @@ Usage: | [exit status] | (see Exit Code table below for values) | `/dev/stdout` | Logs (info) | `/dev/stderr` | Logs (warnings, errors) -| `/.tgz` | Layer snapshot (see [Buildpack Interface Specfication](buildpack.md) -| `/stack-cache.tar.gz` | Cache snaphot for each stack buildpack -| `/stack-layer.toml` | Layer snaphot metadata for each stack buildpack +| `//stack-cache.tgz` | Cache snaphot for each stack buildpack +| `//stack-layer.tgz` | Layer snapshot (see [Buildpack Interface Specfication](buildpack.md) +| `//stack-layer.toml` | Layer snaphot metadata for each stack buildpack | `/config/metadata.toml` | Build metadata (see [`metadata.toml`](#metadatatoml-toml)) | Exit Code | Result| From 439b4ec5ef2c46153c87f48270d2c4ce1e4eff06 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Fri, 18 Dec 2020 10:03:10 -0600 Subject: [PATCH 133/140] Update future work for stackpacks Signed-off-by: Joe Kutner --- text/0000-stack-buildpacks.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index 79ef4d928..f08fd02b0 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -392,7 +392,7 @@ name = "jq" In the future, we plan to enhance the stack buildpack interface with the following: -* A `CNB_STACK_TYPE` env var that a stack buildpack can use to behave differently on each part of the stack +* Tentatively, add a `CNB_STACK_TYPE` env var that a stack buildpack can use to behave differently on each part of the stack * Support for `creator`. Because of the new extend phase, it is not possible to easily run the entire buildpack process with a single binary. * Snapshot caching during the build and extend phase, which would allow stackpack authors cache all changes to the filesystem. From 61c5d734009eadda43dfffe07ff9fcf0b4b9919d Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Fri, 18 Dec 2020 10:04:09 -0600 Subject: [PATCH 134/140] Update text/0000-stack-buildpacks.md Signed-off-by: Joe Kutner Co-authored-by: Emily Casey --- text/0000-stack-buildpacks.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index f08fd02b0..4b9dbab9f 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -35,7 +35,7 @@ A stackpack may also define a list of mixins that it can provide to the stack, o A stack provider may choose to include stack buildpacks with the stack they distribute. If a stack includes a `/cnb/stack/order.toml` file and associated stackpacks, then the following will occur: * Before any lifecycle phases: - 1. The platform may compare the list of mixins that are statically required by all buildpacks (in the `stacks` sections of their `buildpack.toml` files) with the static list of mixins provided by the stack buildpacks (in the `stacks` sections of their `buildpack.toml` files), and fail the build if it chooses to do so. + 1. The platform may compare the list of mixins that are statically required by all buildpacks (in the `stacks` sections of their `buildpack.toml` files) with the static list of mixins provided by the stack images and the stack buildpacks (in the `stacks` sections of their `buildpack.toml` files), and fail the build if the required mixins cannot be provided. * During the detect phase: 1. The lifecycle will compare the list of required mixins to the list of mixins provided by stack and stack buildpacks in accordance with [stage-specific mixin rules](https://github.com/buildpacks/rfcs/pull/109). If any mixins are still not provided, the build will fail. To accomplish this, the list of run-time and build-time mixins that are already present in the run image and build image must be provided to the detector. From 8d94382a98cb025cf1e7941cd1c77ea325877d57 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Fri, 18 Dec 2020 10:05:10 -0600 Subject: [PATCH 135/140] Update text/0000-stack-buildpacks.md Signed-off-by: Joe Kutner Co-authored-by: Emily Casey --- text/0000-stack-buildpacks.md | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index 4b9dbab9f..d2b244b3d 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -86,11 +86,10 @@ A stack can provide stackpacks by including them in the `/cnb/stack/buildpacks` A stackpack will only execute if it passes detection. When the stackpack is executed, its detect and build scripts use the same parameters as the application buildpacks. -The stackpack's snapshot layer may be enriched by writing a `stack-layer.toml` file. The `stack-layer.toml` may define globs of files to be excluded from the image when it is _exported_. Any excluded path may also be marked as _cached_, so that those excluded paths are recovered before the build or extend phase. The term _exported_ is defined as: +The stackpack's snapshot layer may be enriched by writing a `stack-layer.toml` file. The `stack-layer.toml` may define globs of files to be _excluded_ from the image. Any excluded path may also be marked as _cached_, so that the excluded files are recovered before the build or extend phase. The term _excluded_ is defined as: -* *Exported for build-time build*: A given path is excluded at application buildpack build-time, and recovered the next time the build image is extended with the stackpack. -* *Exported for build-time run*: A given path is excluded from the final image, and restored the next time the run image is extended with the stackpack (either rebase or rebuild). -* *Exported for rebase run*: A given path is excluded from the rebased image, and recovered the next time the run image is extended with the stackpack (either rebase or rebuild). +* *Excluded during build phase*: Files at matching paths are removed from the filesystem before application buildpack execution. If cached, on subsequent build operations, the excluded files will be restored to the build-image filesystem before stackpack execution. +* *Excluded during extend phase*: A given path is excluded from the snapshot layer (and thus from the final image). If cached, on subsequent build or rebase operations, the excluded files will be restored to the run-image filesystem before stackpack execution. For example, a stack buildpack may choose to exclude `/var/cache` from the final app image, but may want to mark it as _cached_ to have it restored before the extend phase. From 766ad29587fff1d4ebc9df64c11d2c2ec602d096 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Thu, 14 Jan 2021 09:44:20 -0600 Subject: [PATCH 136/140] Update text/0000-stack-buildpacks.md Signed-off-by: Joe Kutner Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- text/0000-stack-buildpacks.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index d2b244b3d..12fd9a277 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -206,7 +206,7 @@ If the stack does not already provide `google-chrome-stable`, the stackpack may During the export phase, the lifecycle will store any excluded cached stackpack layers created during the build or extend phase in the cache. -During the restore phase of the next build, the lifecycle will download the excluded cached stackpack layers with the cache and store them as tarballs in the `` directory and the `` directory (i.e. they will not be extracted). The `restorer` cannot extract the these layers because it will not run with root privileges. In addition, the `restorer` may run in a different container than the build, which means changes made to the base image are not guaranteed to carry forward. The `builder` will run with root privileges and untar these layers and cache, but will drop privileges after executing stackpacks and before running application buildpacks. The `extender` will also run with root privileges and untar the cache. +During the restore phase of the next build, the lifecycle will download the excluded cached stackpack layers with the cache and store them as tarballs in the `` directory and the `` directory (i.e. they will not be extracted). The `restorer` cannot extract these layers because it will not run with root privileges. In addition, the `restorer` may run in a different container than the build, which means changes made to the base image are not guaranteed to carry forward. The `builder` will run with root privileges and untar these layers and cache, but will drop privileges after executing stackpacks and before running application buildpacks. The `extender` will also run with root privileges and untar the cache. During the build phase, after the stackpacks have run but before the application buildpacks have run, the lifecycle will forcefully delete everything in `[[excludes]]` after ALL stackpacks run, before buildpacks run (which implies that it does not need to do this for extend phase). From 39a683259857485de44fde1bc5eff0477676c799 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Thu, 14 Jan 2021 09:44:33 -0600 Subject: [PATCH 137/140] Update text/0000-stack-buildpacks.md Signed-off-by: Joe Kutner Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- text/0000-stack-buildpacks.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index 12fd9a277..b45ac5580 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -158,8 +158,8 @@ As an example of these resolution rules, consider an Apt stackpack that provide The example stackpack will be skipped after detection in the following cases: * no mixins are required by any component of the build -* A mixin is required by a buildpack, but the stack already provides that mixin. -* A mixin is required by an app, but the stack already provides that mixin. +* a mixin is required by a buildpack, but the stack already provides that mixin +* a mixin is required by an app, but the stack already provides that mixin ### Example: PostgreSQL Certs From d8bb5645b7ec6ef6eb6d10538db5050b1b57c5ea Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Thu, 14 Jan 2021 09:45:16 -0600 Subject: [PATCH 138/140] Update text/0000-stack-buildpacks.md Signed-off-by: Joe Kutner Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- text/0000-stack-buildpacks.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index b45ac5580..b1cd36095 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -152,7 +152,7 @@ A stack buildpack that needs to install mixins must select them from the build p ### Example: Any Mixin Buildpack -As an example of these resolution rules, consider an Apt stackpack that provide any (`*`) mixin. During the detect phase, the stackpack will pass in the following cases: +As an example of these resolution rules, consider an Apt stackpack that provides any (`*`) mixin. During the detect phase, the stackpack will pass in the following cases: * a mixin is required by a buildpack * a mixin is [required by an app](https://github.com/buildpacks/rfcs/pull/112) From d8dc4af623c4c3856b3e19e14b68dd5ddf9038e8 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Thu, 14 Jan 2021 09:49:54 -0600 Subject: [PATCH 139/140] Update text/0000-stack-buildpacks.md Signed-off-by: Joe Kutner Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- text/0000-stack-buildpacks.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index b1cd36095..b6a03d2a8 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -200,7 +200,7 @@ If only the `chromedriver` is required, or the `google-chrome-stable` is already name = "chromedriver" ``` -If the stack does not already provide `google-chrome-stable`, the stackpack may install it anyways. This prevents the case where an app may not work at runtime if that app requires only the `chromedriver` and the `google-chrome-stable` is not provided by the stack. +If the stack does not already provide `google-chrome-stable`, the stackpack may install it anyways, even if the app didn't require it. This prevents the case where an app may not work at runtime if that app requires only the `chromedriver` and the `google-chrome-stable` is not provided by the stack. ## Caching and Restoring From 0da85b3b49ea444372c0c8e82b434e6a714b56e7 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Thu, 14 Jan 2021 09:51:26 -0600 Subject: [PATCH 140/140] Update text/0000-stack-buildpacks.md Signed-off-by: Joe Kutner --- text/0000-stack-buildpacks.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-stack-buildpacks.md b/text/0000-stack-buildpacks.md index b6a03d2a8..da4901ee8 100644 --- a/text/0000-stack-buildpacks.md +++ b/text/0000-stack-buildpacks.md @@ -208,7 +208,7 @@ During the export phase, the lifecycle will store any excluded cached stackpack During the restore phase of the next build, the lifecycle will download the excluded cached stackpack layers with the cache and store them as tarballs in the `` directory and the `` directory (i.e. they will not be extracted). The `restorer` cannot extract these layers because it will not run with root privileges. In addition, the `restorer` may run in a different container than the build, which means changes made to the base image are not guaranteed to carry forward. The `builder` will run with root privileges and untar these layers and cache, but will drop privileges after executing stackpacks and before running application buildpacks. The `extender` will also run with root privileges and untar the cache. -During the build phase, after the stackpacks have run but before the application buildpacks have run, the lifecycle will forcefully delete everything in `[[excludes]]` after ALL stackpacks run, before buildpacks run (which implies that it does not need to do this for extend phase). +During the build phase, after ALL stackpacks have run but before the application buildpacks have run, the lifecycle will forcefully delete everything in `[[excludes]]` (which implies that it does not need to do this for extend phase). ## Rebasing an App