Roadmap: Composable Image-based OS Plans

[pothos]

Flatcar Container Linux is released as generic image (bundling a set of packages in the /usr partition), and as OEM images that supply required additional software in the OEM partition.

As the need for further user customization came up due to particular Docker requirements of early Kubernetes versions, Torcx was added to choose the Docker version or supply a custom one.

However, Torcx wasn't expanded in scope to be more generic, thus the recommended way to deploy custom host-level software was putting binaries in /opt.

When systemd-sysext was announced, we embraced the opportunity for upstream solutions instead of developing our custom OEM software bundling mechanism for OEM updates or continuing with Torcx. Systemd-sysext loads system extension images to overlay their contents on /usr, allowing for a deeper integration than user binaries in /opt or binaries in the OEM partition can provide. It provides a robust method for atomic updates and makes it easy to introspect the current state.

Areas of application: User software, OEMs, Official extensions

Flatcar Stable already provides systemd-sysext as mechanism for user-supplied software (with some caveats). The deprecation of Torcx and the bundling of OEM software as systemd-sysext images are work in progress.

The updating of OEM software will be done together with the /usr partition but supplied as separate update payloads.

Once this mechanism exists, we can also use it to publish official Flatcar extensions that are released as extensions for a particular Flatcar version and covered by the release tests.

For Docker and containerd we might continue to have them on the base image (but as deactivatable systemd-sysext images). Kubernetes binaries or WASM shims, however, could be shipped as such an extension. This mechanism of stackable extensions will also enable us to be more open for alternative container runtimes or other host-level software where the inclusion in /usr of the generic image is hard to justify. It also serves as an alternative to the current small temporary debug tool container aliases we set up, or as a more approachable way to build kernel modules than the developer container in its current form.

In the future certain parts of the current base image could also be moved out to be extensions, so that, e.g, Docker or things like sssd or git would get added at provisioning time.

User-supplied software is not expected to be bound to a particular Flatcar version, and thus it should be built independently from the base OS. Usually this is done by linking against libraries in a custom path or using static linking. The update path is also independent from Flatcar's update mechanisms and can use systemd-sysupdate. While we have a few build scripts that consume upstream static binaries, the building of sysext images from source needs more examples. Hosting of independent sysext images is also an interesting topic for Flatcar because it makes it easier for users to deploy these extensions and update them with systemd-sysupdate. Some overlap with official Flatcar extensions is wanted because it gives users the freedom to choose between auto-updates and explicit control of the version that runs.

Besides systemd-sysext for /usr and /opt, there is systemd-confext for configuration extension images that overlay on /etc. The benefit here as well is a robust update mechanism, and better introspection about what is running.

Upstream work: Mutable overlays

For systemd-sysext to be relevant, it needs use outside of Flatcar. Most Linux distros still need a writable /usr partition which systemd-sysext currently doesn't support. Flatcar also could benefit from writable /opt in case someone wants to use /opt instead of /usr, and Flatcar also would need a mutable overlay with systemd-confext for /etc. As discussed with upstream, there should be different modes, e.g., ephemeral and persistent, but our requirement for /etc is to have /etc as upperdir to provide OS defaults for upcopying. In fact, this mode also makes sense for /usr on traditional distros if they don't want to end up with a split state.

One area of application could be CAPI sysext images that also work well on Ubuntu and finally provide an in-place update mechanism for CAPI nodes.

Milestones

• OEM updates for all images
• Consumption/updates of user-supplied software from the sysext-bakery or built from the SDK
• Torcx removal, internal Docker+containerd sysext images (could be moved out later, but enabled by default)
• Podman extension
• WASM extension (runwasi shims and standalone wasmtime)
• Kata container extension
• Hypervisor extension: QEMU and Cloud Hypervisor - maybe also kata ↑
• ZFS extension (prebuilt modules)
• Hypervisor extension
• Nvidia extension (Open Source kernel module, CRI integration)
• Developer extension (same packages as the Flatcar devcontainer – maybe even debug symbols, or in their own extension)
• Split out of extra tools from the base image to "extras" extension (enabled by default)
• Kubernetes/CAPI extension, both as Flatcar extension and independent
• Improved systemd-sysext/systemd-confext support

[pothos]

I think we can also have a "hypervisor" extension that ships qemu and cloud hypervisor.
When using cloud hypervisor it's actually good to have qemu-img around for VM image operations.
The extension could also provide the binfmt files for binary translation, i.e., running amd64 containers on arm64 or vice versa (also common when building software).

Edit: This could also include the kata-containers package.