Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move analyzer before detector #197

Merged
merged 15 commits into from
Apr 29, 2021
+118 −102
Merged

Move analyzer before detector #197

Changes from 4 commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
df6ac6f
Move analyzer before detector
jabrown85 Feb 24, 2021
326541a
PR Changes
jabrown85 Feb 25, 2021
4cdddd7
added analyzed path to restorer
jabrown85 Feb 25, 2021
83178a2
Added more fields from the RFC
jabrown85 Mar 3, 2021
d0e9221
Removing fields to put into another spec PR
jabrown85 Mar 11, 2021
f1ab6a5
Apply suggestions from code review
jabrown85 Mar 12, 2021
449ce31
More PR changes
jabrown85 Mar 12, 2021
c8b108e
adding back all the things missing from RFC for review
jabrown85 Mar 22, 2021
ae933a8
Updated spec PR to align with latest discussions.
jabrown85 Mar 29, 2021
6bda2a6
Apply suggestions from code review
jabrown85 Mar 31, 2021
323cb33
Apply suggestions from code review
jabrown85 Apr 12, 2021
05aa33a
Update platform.md
jabrown85 Apr 22, 2021
a3be6e5
Added analyzed to detector inputs
jabrown85 Apr 22, 2021
dd17824
Update platform.md
jabrown85 Apr 28, 2021
5a9efbb
Merge branch 'platform/0.7' into patch-2
jkutner Apr 29, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
167 changes: 87 additions & 80 deletions platform.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,17 +32,16 @@ Examples of a platform might include:
- [Rebase](#rebase)
- [Launch](#launch)
- [Usage](#usage)
- [`detector`](#detector)
- [`analyzer`](#analyzer)
- [Inputs](#inputs)
- [Outputs](#outputs)
- [`analyzer`](#analyzer)
- [`detector`](#detector)
- [Inputs](#inputs-1)
- [Outputs](#outputs-1)
- [Layer analysis](#layer-analysis)
- [`restorer`](#restorer)
- [Inputs](#inputs-2)
- [Outputs](#outputs-2)
- [Layer restoration](#layer-restoration)
- [Layer Restoration](#layer-restoration)
- [`builder`](#builder)
- [Inputs](#inputs-3)
- [Outputs](#outputs-3)
Expand Down Expand Up @@ -177,7 +176,7 @@ The platform SHOULD ensure that:
- The image config's `Label` field has the label `io.buildpacks.stack.distro.version` set to the version of the stack's OS distro.
- The image config's `Label` field has the label `io.buildpacks.stack.released` set to the release date of the stack.
- The image config's `Label` field has the label `io.buildpacks.stack.description` set to the description of the stack.
- The image config's `Label` field has the label `io.buildpacks.stack.metadata` set to additional metadata related to the stack.
- The image config's `Label` field has the label `io.buildpacks.stack.metadata` set to additional metadata related to the stack.

### Run Image

Expand All @@ -196,7 +195,7 @@ The platform SHOULD ensure that:
- The image config's `Label` field has the label `io.buildpacks.stack.distro.version` set to the version of the stack's OS distro.
- The image config's `Label` field has the label `io.buildpacks.stack.released` set to the release date of the stack.
- The image config's `Label` field has the label `io.buildpacks.stack.description` set to the description of the stack.
- The image config's `Label` field has the label `io.buildpacks.stack.metadata` set to additional metadata related to the stack.
- The image config's `Label` field has the label `io.buildpacks.stack.metadata` set to additional metadata related to the stack.

### Mixins

Expand Down Expand Up @@ -237,15 +236,15 @@ If `CNB_PLATFORM_API` is set in the lifecycle's execution environment, the lifec
#### Build
A single app image build* consists of the following phases:

1. Detection
1. Analysis
1. Detection
1. Cache Restoration
1. Build*
1. Export

A platform MUST execute these phases either by invoking the following phase-specific lifecycle binaries in order:
1. `/cnb/lifecycle/detector`
1. `/cnb/lifecycle/analyzer`
1. `/cnb/lifecycle/detector`
1. `/cnb/lifecycle/restorer`
1. `/cnb/lifecycle/builder`
1. `/cnb/lifecycle/exporter`
Expand All @@ -264,7 +263,7 @@ This entire operation is referred to as rebasing the app image.
Rebasing allows for fast runtime OS-level dependency updates for app images without requiring a rebuild. A rebase requires minimal data transfer when the app and run images are colocated on a Docker registry that supports [Cross Repository Blob Mounts](https://docs.docker.com/registry/spec/api/#cross-repository-blob-mount).

To rebase an app image a platform MUST execute the `/cnb/lifecycle/rebaser` or perform an equivalent operation.

#### Launch
`/cnb/lifecycle/launcher` is responsible for launching user and buildpack provided processes in the correct execution environment.
`/cnb/lifecycle/launcher` SHALL be the `ENTRYPOINT` for all app images.
Expand All @@ -276,10 +275,69 @@ All lifecycle phases:
- MUST read `CNB_PLATFORM_API` from the execution environment and evaluate compatibility before attempting to parse other inputs (see [Platform API Compatibility](#platform-api-compatibility))
- MUST give command line inputs precedence over other inputs

#### `analyzer`
Usage:
```
/cnb/lifecycle/analyzer \
[-analyzed <analyzed>] \
[-cache-dir <cache-dir>] \
[-cache-image <cache-image>] \
[-daemon] \ # sets <daemon>
[-gid <gid>] \
[-group <group>] \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From the lifecycle PR: it looks like group is only defined for earlier platform apis, is that intended?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that I'm remembering how all of this works... we can't have group if analyze goes first...

Suggested change
[-group <group>] \

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can have group, but not order

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can't have group right? There is no group until detection.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the future we will support stack-group in order to validate mixins, but not group?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I confused myself with the names. We need <order>, not <group>. Fixed in 6480d2e

[-layers <layers>] \
[-log-level <log-level>] \
[-previous-image <previous-image> ] \
[-run-image <run-image> ] \
[-stack-id <stack-id> ] \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have never accepted -stack-id as a flag before. Since we added the build-image stack ID to stack.toml do we need to provide an override here. It seems like we should use <stack> instead.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed in 6480d2e

[-stack <stack> ] \
[-uid <uid>]
```

##### Inputs
| Input | Environment Variable | Default Value | Description
|-------------------|-----------------------|--------------------------|----------------------
| `<analyzed>` | `CNB_ANALYZED_PATH` | `<layers>/analyzed.toml` | Path to output analysis metadata (see [`analyzed.toml`](#analyzedtoml-toml)
| `<cache-dir>` | `CNB_CACHE_DIR` | | Path to a cache directory
| `<cache-image>` | `CNB_CACHE_IMAGE` | | Location of cache, provided as an image
| `<daemon>` | `CNB_USE_DAEMON` | `false` | Analyze image from docker daemon
| `<gid>` | `CNB_GROUP_ID` | | Primary GID of the stack `User`
| `<group>` | `CNB_GROUP_PATH` | `<layers>/group.toml` | Path to group definition (see [`group.toml`](#grouptoml-toml))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
| `<group>` | `CNB_GROUP_PATH` | `<layers>/group.toml` | Path to group definition (see [`group.toml`](#grouptoml-toml))

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed in 6480d2e

| `<layers>` | `CNB_LAYERS_DIR` | `/layers` | Path to layers directory
| `<log-level>` | `CNB_LOG_LEVEL` | `info` | Log Level
| `<previous-image>`| `CNB_PREVIOUS_IMAGE` | `<image>` | Image reference to be analyzed (usually the result of the previous build)
| `<run-image>` | `CNB_RUN_IMAGE` | resolved from <stack> | Run image reference
| `<stack-id>` | `CNB_STACK_ID` | | Path to stack file (see [`stack.toml`](#stacktoml-toml))
| `<stack>` | `CNB_STACK_PATH` | `/cnb/stack.toml` | Chosen stack ID
| `<uid>` | `CNB_USER_ID` | | UID of the stack `User`

- **If** `<daemon>` is `false`, `<image>` MUST be a valid image reference
- **If** `<daemon>` is `true`, `<image>` MUST be either a valid image reference or an imageID
- **If** `<run-image>` is not provided by the platform the value will be resolved from the contents of stack
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- **If** `<run-image>` is not provided by the platform the value will be resolved from the contents of stack
- **If** `<run-image>` is not provided by the platform the lifecycle MUST [resolve](#run-image-resolution) the run image from the contents of `stack` or fail if `stack` does not contain a valid run image.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in 6480d2e

- The lifecycle MUST accept valid references to non-existent images without error.

##### Outputs
| Output | Description
|--------------------|----------------------------------------------
| [exit status] | (see Exit Code table below for values)
| `/dev/stdout` | Logs (info)
| `/dev/stderr` | Logs (warnings, errors)
| `<analyzed>` | Analysis metadata (see [`analyzed.toml`](#analyzedtoml-toml)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that we are writing run-image metadata in this output. analyzed.toml needs to be an input to exporter and stack.toml can be removed as an exporter input.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

<analyzed> is already an input, I removed <stack> in 6480d2e


| Exit Code | Result|
|-----------|-------|
| `0` | Success
| `11` | Platform API incompatibility error
| `12` | Buildpack API incompatibility error
| `1-10`, `13-99` | Generic lifecycle errors
| `200-299` | Analysis-specific lifecycle errors

- The lifecycle MUST write [analysis metadata](#analyzedtoml-toml) to `<analyzed>` if `<image>` is accessible.

#### `detector`
The platform MUST execute `detector` in the **build environment**

Usage:
Usage:
```
/cnb/lifecycle/detector \
[-app <app>] \
Expand Down Expand Up @@ -327,65 +385,57 @@ The lifecycle:
- SHALL detect a single group from `<order>` and write it to `<group>` using the [detection process](buildpack.md#phase-1-detection) outlined in the Buildpack Interface Specification
- SHALL write the resolved build plan from the detected group to `<plan>`

#### `analyzer`
Usage:
#### `restorer`
Usage:
```
/cnb/lifecycle/analyzer \
/cnb/lifecycle/restorer \
[-analyzed <analyzed>] \
[-cache-dir <cache-dir>] \
[-cache-image <cache-image>] \
[-daemon] \ # sets <daemon>
[-gid <gid>] \
[-group <group>] \
[-layers <layers>] \
[-log-level <log-level>] \
[-skip-layers <skip-layers>] \
[-uid <uid>] \
<image>
[-uid <uid>]
```

##### Inputs
| Input | Environment Variable | Default Value | Description
|----------------|-----------------------|--------------------------|----------------------
| `<analyzed>` | `CNB_ANALYZED_PATH` | `<layers>/analyzed.toml` | Path to output analysis metadata (see [`analyzed.toml`](#analyzedtoml-toml)
| `<cache-dir>` | `CNB_CACHE_DIR` | | Location of cache, provided as a directory
| `<cache-image>`| `CNB_CACHE_IMAGE` | | Location of cache, provided as an image
| `<daemon>` | `CNB_USE_DAEMON` | `false` | Analyze image from docker daemon
| `<analyzed>` | `CNB_ANALYZED_PATH` | `<layers>/analyzed.toml` | Path to analysis metadata (see [`analyzed.toml`](#analyzedtoml-toml)
| `<cache-dir>` | `CNB_CACHE_DIR` | | Path to a cache directory
| `<cache-image>`| `CNB_CACHE_IMAGE` | | Reference to a cache image in an OCI image registry
| `<gid>` | `CNB_GROUP_ID` | | Primary GID of the stack `User`
| `<group>` | `CNB_GROUP_PATH` | `<layers>/group.toml` | Path to group definition (see [`group.toml`](#grouptoml-toml))
| `<image>` | | | Image reference to be analyzed (usually the result of the previous build)
| `<layers>` | `CNB_LAYERS_DIR` | `/layers` | Path to layers directory
| `<log-level>` | `CNB_LOG_LEVEL` | `info` | Log Level
| `<skip-layers>`| `CNB_SKIP_LAYERS` | `false` | Do not perform layer analysis
| `<uid>` | `CNB_USER_ID` | | UID of the stack `User`

- **If** `<daemon>` is `false`, `<image>` MUST be a valid image reference
- **If** `<daemon>` is `true`, `<image>` MUST be either a valid image reference or an imageID
- The lifecycle MUST accept valid references to non-existent images without error.
| `<skip-layers>`| `CNB_SKIP_LAYERS` | `false` | Do not perform layer analysis

##### Outputs
| Output | Description
|--------------------|----------------------------------------------
| [exit status] | (see Exit Code table below for values)
| `/dev/stdout` | Logs (info)
| `/dev/stderr` | Logs (warnings, errors)
| `<analyzed>` | Analysis metadata (see [`analyzed.toml`](#analyzedtoml-toml)
| Output | Description
|---------------------------------------|----------------------------------------------
| [exit status] | (see Exit Code table below for values)
| `/dev/stdout` | Logs (info)
| `/dev/stderr` | Logs (warnings, errors)
| `<layers>/<buidpack-id>/<layer>.sha` | Files containing the diffID of each analyzed layer
| `<layers>/<buidpack-id>/<layer>.toml` | Files containing the layer content metadata of each analyzed layer (see data format in [Buildpack Interface Specification](buildpack.md))
| `<layers>/<buidpack-id>/<layer>/*`. | Restored layer contents

| Exit Code | Result|
|-----------|-------|
| `0` | Success
| `11` | Platform API incompatibility error
| `12` | Buildpack API incompatibility error
| `1-10`, `13-99` | Generic lifecycle errors
| `200-299` | Analysis-specific lifecycle errors
| `300-399` | Restoration-specific lifecycle errors

- The lifecycle MUST write [analysis metadata](#analyzedtoml-toml) to `<analyzed>` if `<image>` is accessible.
- **If** `<skip-layers>` is `true` the lifecycle MUST NOT perform layer analysis.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- **If** `<skip-layers>` is `true` the lifecycle MUST NOT perform layer analysis.
- For each buildpack in `<group>`, if persistent metadata for that buildpack exists in the analysis metadata, lifecycle MUST write a toml representation of the persistent metadata to `<layers>/<buildpack-id>/store.toml`
- **If** `<skip-layers>` is `true` the lifecycle MUST NOT perform layer analysis.

We also need to modify the data format for io.buildpacks.lifecycle.metadata[https://github.com/buildpacks/spec/blob/main/platform.md#iobuildpackslifecyclemetadata-json] to included persistent metadata in the format currently expected by the lifecycle.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What has changed here? Sorry, I don't see what needs to be modified.

- **Else** the lifecycle MUST analyze any app image layers or cached layers created by any buildpack present in the provided `<group>`.

##### Layer analysis

##### Layer Restoration
When analyzing a given layer the lifecycle SHALL:
- **If** `build=true`, `cache=false`:
- Do nothing
Expand All @@ -396,57 +446,14 @@ When analyzing a given layer the lifecycle SHALL:
- Write layer metadata read from the cache to `<layers>/<buildpack-id>/<layer-name>.toml`
- Write the layer diffID from the cache to `<layers>/<buildpack-id>/<layer-name>.sha`

#### `restorer`
Usage:
```
/cnb/lifecycle/restorer \
[-cache-dir <cache-dir>]
[-cache-image <cache-image>]
[-gid <gid>] \
[-group <group>] \
[-layers <layers>] \
[-log-level <log-level>] \
[-uid <uid>]
```

##### Inputs
| Input | Environment Variable | Default Value | Description
|----------------|-----------------------|-----------------------|----------------------
| `<cache-dir>` | `CNB_CACHE_DIR` | | Path to a cache directory
| `<cache-image>`| `CNB_CACHE_IMAGE` | | Reference to a cache image in an OCI image registry
| `<gid>` | `CNB_GROUP_ID` | | Primary GID of the stack `User`
| `<group>` | `CNB_GROUP_PATH` | `<layers>/group.toml` | Path to group definition (see [`group.toml`](#grouptoml-toml))
| `<layers>` | `CNB_LAYERS_DIR` | `/layers` | Path to layers directory
| `<log-level>` | `CNB_LOG_LEVEL` | `info` | Log Level
| `<uid>` | `CNB_USER_ID` | | UID of the stack `User`
| `<layers>/<buidpack-id>/<layer>.sha` || | Files containing the diffID of each analyzed layer
| `<layers>/<buidpack-id>/<layer>.toml` || | Files containing the layer content metadata of each analyzed layer (see data format in [Buildpack Interface Specification](buildpack.md))

##### Outputs
| Output | Description
|------------------------------------|----------------------------------------------
| [exit status] | (see Exit Code table below for values)
| `/dev/stdout` | Logs (info)
| `/dev/stderr` | Logs (warnings, errors)
| `<layers>/<buidpack-id>/<layer>/*` | Restored layer contents

| Exit Code | Result|
|-----------|-------|
| `0` | Success
| `11` | Platform API incompatibility error
| `12` | Buildpack API incompatibility error
| `1-10`, `13-99` | Generic lifecycle errors
| `300-399` | Restoration-specific lifecycle errors

##### Layer restoration
For each layer metadata file found in the `<layers>` directory, the lifecycle:
- MUST restore cached layer contents if the cache contains a layer with matching diffID
- MUST remove layer metadata if `cache=true` AND the cache DOES NOT contain a layer with matching diffID

#### `builder`
The platform MUST execute `builder` in the **build environment**

Usage:
Usage:
```
/cnb/lifecycle/builder \
[-app <app>] \
Expand Down Expand Up @@ -808,7 +815,7 @@ The following variables SHOULD be set in the lifecycle execution environment and
|-----------------|--------------------------------------
| `CNB_STACK_ID` | Chosen stack ID
| `HOME` | Current user's home directory

The following variables SHOULD be set in the lifecycle execution environment and MAY be modified by prior buildpacks before they are provided to a given buildpack:

| Env Variable | Layer Path | Contents
Expand Down