From b2ec69f6b8945b003c2e8b488261df5b1152cac8 Mon Sep 17 00:00:00 2001 From: Colin Walters Date: Tue, 19 Oct 2021 17:39:03 -0400 Subject: [PATCH 1/5] enhancements: Propose CoreOS layering See text for rationale. --- os/coreos-layering.md | 76 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 76 insertions(+) create mode 100644 os/coreos-layering.md diff --git a/os/coreos-layering.md b/os/coreos-layering.md new file mode 100644 index 00000000..3e9f5041 --- /dev/null +++ b/os/coreos-layering.md @@ -0,0 +1,76 @@ +# CoreOS Layering + +This enhancement proposes: + +- A fundamental new addition to ostree/rpm-ostree, which is support for directly pulling and updating the OS from container images (while keeping all existing functionality, per-node package layering and our existing support for pulling via ostree on the wire). +- Documentation for generating derived (layered) images from the pristine CoreOS base image. +- Support for switching a FCOS system to use a custom image on firstboot via Ignition +- zincati will continue to perform upgrades by inspecting the upgrade graph from the base image. + +# Existing work + +Originally, https://github.com/coreos/fedora-coreos-tracker/issues/812 tracked native support for "encapsulating" ostree commits in containers. + +Then, it was realized that when shipping the OS as a container image, it feels natural to support users deriving from it. The bulk of this is really "ostree native container" integration glue, and is happening in https://github.com/ostreedev/ostree-rs-ext + +rpm-ostree vendors the ostree-rs-ext code and also be extended to support the same interfaces as implemented by the "base" ostree-rs-ext code. + +Specifically as of today, this functionality is exposed in: + +- `rpm-ostree ex-container` +- `rpm-ostree rebase --experimental $containerref` + +# Rationale + +Since the creation of Container Linux (CoreOS) as well as Atomic Host, and continuing into the Fedora/RHEL CoreOS days, we have faced a constant tension around what we ship in the host system. + +[This issue](https://github.com/coreos/fedora-coreos-tracker/issues/401) encapsulates much prior discussion. + +For users who are happy with Fedora CoreOS today, not much will change. + +For those who e.g. want to install custom agents or nontrivial amounts of code (such as kubelet), this "CoreOS layering" will be a powerful new mechanism to ship the code they need. + +# Example + +[fcos-derivation-example](https://github.com/cgwalters/fcos-derivation-example) contains an example `Dockerfile` that builds a Go binary and injects it along with a corresponding systemd unit as a layer, building on top of Fedora CoreOS. + +For ease of reference, a copy of the above is inline here: + +```dockerfile +# Build a small Go program using a builder image +FROM registry.access.redhat.com/ubi8/ubi:latest as builder +WORKDIR /build +COPY . . +RUN yum -y install go-toolset +RUN go build hello-world.go + +# In the future, this would be e.g. quay.io/coreos/fedora:stable +FROM quay.io/cgwalters/fcos-dev +# Inject it into Fedora CoreOS +COPY --from=builder /build/hello-world /usr/bin +# And add our unit file +ADD hello-world.service /etc/systemd/system/hello-world.service +# Also add strace; we don't yet support `yum install` but we can +# with some work in rpm-ostree! +RUN rpm -Uvh https://kojipkgs.fedoraproject.org//packages/strace/5.14/1.fc34/x86_64/strace-5.14-1.fc34.x86_64.rpm +``` + +# Derivation versus Ignition/butane + +This proposal does not replace Ignition. Ignition will still play two key roles: + +- Setting up partitions and storage, e.g. LUKS is something configured via Ignition provided on boot. +- Machine/node specific configuration, in particular bootstrap configuration: e.g. static IP addresses that are necessary to fetch container images at all. +# Use of CoreOS disk/boot images + +And more explicitly, it's expected that many if not most users would continue to use the official Fedora CoreOS "boot images" (e.g. ISO, AMI, qcow2, etc.). This proposal does *not* currently call for exposing a way for a user to create the boot image shell around their custom container, although that is an obvious potential next step. + +Hence, a user wanting to use a custom base image would provide machines with an Ignition config that performs e.g. `rpm-ostree rebase ostree-remote-image:quay.io/examplecorp/baseos:latest` as a systemd unit. It is likely that we would provide this via [butane](github.com/coreos/butane) as well; for example: + +``` +variant: fcos +version: 1.1.0 +ostree_container: + image: quay.io/mycorp/myfcos:stable + reboot: true +``` From 1fe203bd7525ff9458071d7181ad1b9b79245da9 Mon Sep 17 00:00:00 2001 From: Colin Walters Date: Tue, 16 Nov 2021 16:40:06 -0500 Subject: [PATCH 2/5] Talk about butane as declarative format --- os/coreos-layering.md | 34 +++++++++++++++++++++++++++++----- 1 file changed, 29 insertions(+), 5 deletions(-) diff --git a/os/coreos-layering.md b/os/coreos-layering.md index 3e9f5041..c6e37444 100644 --- a/os/coreos-layering.md +++ b/os/coreos-layering.md @@ -30,7 +30,7 @@ For users who are happy with Fedora CoreOS today, not much will change. For those who e.g. want to install custom agents or nontrivial amounts of code (such as kubelet), this "CoreOS layering" will be a powerful new mechanism to ship the code they need. -# Example +# Example via Dockerfile [fcos-derivation-example](https://github.com/cgwalters/fcos-derivation-example) contains an example `Dockerfile` that builds a Go binary and injects it along with a corresponding systemd unit as a layer, building on top of Fedora CoreOS. @@ -55,21 +55,45 @@ ADD hello-world.service /etc/systemd/system/hello-world.service RUN rpm -Uvh https://kojipkgs.fedoraproject.org//packages/strace/5.14/1.fc34/x86_64/strace-5.14-1.fc34.x86_64.rpm ``` -# Derivation versus Ignition/butane +# Derivation versus Ignition/Butane -This proposal does not replace Ignition. Ignition will still play two key roles: +This proposal does not replace Ignition. Ignition will still play at least two key roles: - Setting up partitions and storage, e.g. LUKS is something configured via Ignition provided on boot. - Machine/node specific configuration, in particular bootstrap configuration: e.g. static IP addresses that are necessary to fetch container images at all. + +# Butane as a declarative input format for layering + +We must support `Dockerfile`, because it's the lowest common denominator for the container ecosystem, and is accepted as input for many tools. + +However, one does not have to use `Dockerfile` to make containers. Specifically, what would make a lot of sense for Fedora CoreOS is to focus +on Butane as a standard declarative interface to this process. + +This could run as a "builder" container, something like this: + +``` +FROM quay.io/coreos/butane:release +COPY . . +RUN butane -o /build/ignition.json + +FROM quay.io/fedora/coreos:stable +COPY --from=builder /build/ignition.json /tmp/ +RUN ignition --write-filesystem /tmp/ignition.json && rm -f /tmp/ignition.json +``` + +Another option is to support being run nested inside an existing container tool, similar to +[kaniko](https://github.com/GoogleContainerTools/kaniko). Then no +`Dockerfile` would be needed. + # Use of CoreOS disk/boot images And more explicitly, it's expected that many if not most users would continue to use the official Fedora CoreOS "boot images" (e.g. ISO, AMI, qcow2, etc.). This proposal does *not* currently call for exposing a way for a user to create the boot image shell around their custom container, although that is an obvious potential next step. -Hence, a user wanting to use a custom base image would provide machines with an Ignition config that performs e.g. `rpm-ostree rebase ostree-remote-image:quay.io/examplecorp/baseos:latest` as a systemd unit. It is likely that we would provide this via [butane](github.com/coreos/butane) as well; for example: +Hence, a user wanting to use a custom base image would provide machines with an Ignition config that performs e.g. `rpm-ostree rebase ostree-remote-image:quay.io/examplecorp/baseos:latest` as a systemd unit. It is likely that we would provide this via [Butane](github.com/coreos/butane) as well; for example: ``` variant: fcos -version: 1.1.0 +version: 1.5.0 ostree_container: image: quay.io/mycorp/myfcos:stable reboot: true From b7608f825ffba4e08d21066a94a9db0991e012ca Mon Sep 17 00:00:00 2001 From: Colin Walters Date: Thu, 18 Nov 2021 17:58:06 -0500 Subject: [PATCH 3/5] Add blurb about Dockerfile --- os/coreos-layering.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/os/coreos-layering.md b/os/coreos-layering.md index c6e37444..11d9bcf1 100644 --- a/os/coreos-layering.md +++ b/os/coreos-layering.md @@ -32,6 +32,12 @@ For those who e.g. want to install custom agents or nontrivial amounts of code ( # Example via Dockerfile +One goal the current CoreOS team has in this is to still preserve the distinction we have between "base image" and user content. +This argues for a declarative input that only allows controlled mutation. + +However, `Dockerfile` is the lowest common denominator in the container ecosystem. +To truly illustrate the goal of supporting arbitrary inputs, we must support it. + [fcos-derivation-example](https://github.com/cgwalters/fcos-derivation-example) contains an example `Dockerfile` that builds a Go binary and injects it along with a corresponding systemd unit as a layer, building on top of Fedora CoreOS. For ease of reference, a copy of the above is inline here: From 03e25faf258168dd05555d6729d708efa0c159df Mon Sep 17 00:00:00 2001 From: Colin Walters Date: Mon, 22 Nov 2021 10:17:54 -0500 Subject: [PATCH 4/5] Update os/coreos-layering.md Co-authored-by: Jonathan Lebon --- os/coreos-layering.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/os/coreos-layering.md b/os/coreos-layering.md index 11d9bcf1..9e259687 100644 --- a/os/coreos-layering.md +++ b/os/coreos-layering.md @@ -33,7 +33,7 @@ For those who e.g. want to install custom agents or nontrivial amounts of code ( # Example via Dockerfile One goal the current CoreOS team has in this is to still preserve the distinction we have between "base image" and user content. -This argues for a declarative input that only allows controlled mutation. +This argues for a declarative input that only allows controlled mutation. See the next section about this. However, `Dockerfile` is the lowest common denominator in the container ecosystem. To truly illustrate the goal of supporting arbitrary inputs, we must support it. From 1bcd3b528caf3b289ba8d2da571e79694a91fab1 Mon Sep 17 00:00:00 2001 From: Colin Walters Date: Mon, 22 Nov 2021 10:18:04 -0500 Subject: [PATCH 5/5] Update os/coreos-layering.md Co-authored-by: Jonathan Lebon --- os/coreos-layering.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/os/coreos-layering.md b/os/coreos-layering.md index 9e259687..4eeab3a1 100644 --- a/os/coreos-layering.md +++ b/os/coreos-layering.md @@ -61,6 +61,13 @@ ADD hello-world.service /etc/systemd/system/hello-world.service RUN rpm -Uvh https://kojipkgs.fedoraproject.org//packages/strace/5.14/1.fc34/x86_64/strace-5.14-1.fc34.x86_64.rpm ``` +# Controlled mutation + +One of the primary advantages of the `Dockerfile` layering approach is that it allows direct filesystem modifications. However, we should distinguish between layering things (e.g. `/etc` files, or third-party daemon) and modifying base content (e.g. fast-track kernel hotfix), which has a higher likelihood of invalidating our CI process. In a cluster context for example, it's possible that a cluster admin may want to permit users only certain modifications. + +It is expected that control mechanisms will be integrated, though it's still not clear how that will look. It may be inside rpm-ostree (e.g. requiring override switches when first rebasing to the pullspec, and/or requiring a specific label on the image), or as part of the image build process itself (e.g. as part of [finalization](https://github.com/ostreedev/ostree-rs-ext/issues/159)). Of course, higher-level interfaces may enforce even stricter guidelines or only accept easily verifiable configs such as Butane/Ignition (see Butane example below). + +Ideally, it shouldn't be difficult for an FCOS/RHCOS user to query the kinds of mutations inside a container image, and this could then be displayed in a succinct way as part of `rpm-ostree status` when rebased onto it. # Derivation versus Ignition/Butane This proposal does not replace Ignition. Ignition will still play at least two key roles: