forked from gardener/gardener
-
Notifications
You must be signed in to change notification settings - Fork 1
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
🕵️ Introduce lib for
gardener-node-agent (gardener#8249)
* Introduce lib for gardener-node-agent gardener#8023 * Apply suggestions from code review Co-authored-by: Johannes Scheerer <johannes.scheerer@sap.com> * Revisit first parts of the node agent concept * Rephrase reason comparision * Speed benefits mostly in large clusters * Remove commented imports * refactor(nodeagent): rename extractTarGz * fix(nodeagent): pick newest file from layers * fix(nodeagent): dropped projected info for token That the token is projected doesn't matter. * fix(nodeagent): removed empty tests Nothing to test here * fix(nodeagent): mirror v1alpha1 changes * refactor(nodeagent): dbus logs, events and naming * feat(nodeagent): validate for supported kubernetesversion * feat(nodeagent): improved coverage for config validation * revert(nodeagent): fake dbus tests did not provide any value * docs(nodeagent): fix config registration docs * fix(docs): reorder the basic design and postpone installation * docs(nodeagent): binary path * docs(nodeagent): be more explicit between cloud config and osc * docs(nodeagent): link operatingsystemconfig extension * docs(nodeagent): future development section Removes the TODO inside the Scalability section and appends it in a separate section. * fix(codegen): generate nodeagent * fix(nodeagent): fix checks * Update pkg/nodeagent/apis/config/validation/validation_test.go Co-authored-by: Oliver Götz <47362717+oliver-goetz@users.noreply.github.com> * docs(nodeagent): rename architecture svg * docs(nodeagent): improved wording * fix(nodeagent): camel cased validation * fix(nodeagent): wording * docs(nodeagent): prefer `kubelet` * docs(nodeagent): `kube-apiserver` * fix(nodeagent): validation test specs * fix(dbus): remove empty suite * refactor(dbus): typo and formatting * fix(nodeagent): extract secure from remote * Apply suggestions from code review Co-authored-by: Rafael Franzke <rafael.franzke@sap.com> * docs(nodeagent): rephrase gardener community * docs(nodeagent): remove mentioning of supported archs * refactor(nodeagent): rename api types * fix(nodeagent): lowercase kubelet data volume size * refactor(nodeagent): validation naming and formatting * refactor(nodeagent): binary cabundle * refactor(nodeagent): use semver for Kubernetes Version * fix(nodeagent): remove unused fake dbus Currently it is unused. In an upcoming PR it will be reintroduced by future controllers. * Apply suggestions from code review Co-authored-by: Johannes Scheerer <johannes.scheerer@sap.com> * tmp: delete controller-registration to regenerate * chore: generate * docs(nodeagent): corrected architecture diagram * revert: controller registration due to tar incompatabilities * docs(nodeagent): wording architecture diagram * fix(generate): add trailing newline for controller registration * feat(nodeagent): test registry extraction * fix(nodeagent): lint * Update pkg/nodeagent/apis/config/validation/validation.go Co-authored-by: Johannes Scheerer <johannes.scheerer@sap.com> * PR review feedback --------- Co-authored-by: Johannes Scheerer <johannes.scheerer@sap.com> Co-authored-by: Oliver Götz <47362717+oliver-goetz@users.noreply.github.com> Co-authored-by: Rafael Franzke <rafael.franzke@sap.com>
- Loading branch information
4 people
authored
Oct 4, 2023
1 parent
ba583aa
commit b485706
Showing
375 changed files
with
71,558 additions
and
462 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
310 changes: 310 additions & 0 deletions
310
docs/concepts/images/gardener-nodeagent-architecture.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,63 @@ | ||
| # Gardener Node Agent | ||
|
|
||
| The goal of the `gardener-node-agent` is to bootstrap a machine into a worker node and maintain node-specific components, which run on the node and are unmanaged by Kubernetes (e.g. the `kubelet` service, systemd units, ...). | ||
|
|
||
| It effectively is a Kubernetes controller deployed onto the worker node. | ||
|
|
||
| ## Architecture and Basic Design | ||
|
|
||
|  | ||
|
|
||
| This figure visualizes the overall architecture of the `gardener-node-agent`. On the left side, it starts with an [`OperatingSystemConfig`](../extensions/operatingsystemconfig.md) resource (`OSC`) with a corresponding worker pool specific `cloud-config-<worker-pool>` secret being passed by reference through the userdata to a machine by the `machine-controller-manager` (MCM). | ||
|
|
||
| On the right side, the `cloud-config` secret will be extracted and used by the `gardener-node-agent` after being installed. Details on this can be found in the next section. | ||
|
|
||
| Finally, the `gardener-node-agent` runs a systemd service watching on secret resources located in the `kube-system` namespace like our `cloud-config` secret that contains the `OperatingSystemConfig`. When `gardener-node-agent` applies the OSC, it installs the `kubelet` + configuration on the worker node. | ||
|
|
||
| ## Installation and Bootstrapping | ||
|
|
||
| This section describes how the `gardener-node-agent` is initially installed onto the worker node. | ||
|
|
||
| In the beginning, there is a very small bash script called [`gardener-node-init.sh`](../../pkg/component/extensions/operatingsystemconfig/original/components/containerd/templates/scripts/init.tpl.sh), which will be copied to `/var/lib/gardener-node-agent/gardener-node-init.sh` on the node with cloud-init data. This script's sole purpose is downloading and starting the `gardener-node-agent`. The binary artifact is extracted from an [OCI artifact](https://github.com/opencontainers/image-spec/blob/main/manifest.md) and lives at `/usr/local/bin/gardener-node-agent`. The `kubelet` should also be contained in the same OCI artifact. | ||
|
|
||
| Along with the init script, a configuration for the `gardener-node-agent` is carried over to the worker node at `/var/lib/gardener-node-agent/configuration.yaml`. This configuration contains things like the shoot's `kube-apiserver` endpoint, the according certificates to communicate with it, the bootstrap token for the `kubelet`, and so on. | ||
|
|
||
| In a bootstrapping phase, the `gardener-node-agent` sets itself up as a systemd service. It also executes tasks that need to be executed before any other components are installed, e.g. formatting the data device for the `kubelet`. | ||
|
|
||
| ## Reasoning | ||
|
|
||
| The `gardener-node-agent` is a replacement for what was called the `cloud-config-downloader` and the `cloud-config-executor`, both written in `bash`. The `gardener-node-agent` implements this functionality as a regular controller and feels more uniform in terms of maintenance. | ||
|
|
||
| With the new architecture we gain a lot, let's describe the most important gains here. | ||
|
|
||
| ### Developer Productivity | ||
|
|
||
| Since the Gardener community develops in Go day by day, writing business logic in `bash` is difficult, hard to maintain, almost impossible to test. Getting rid of almost all `bash` scripts which are currently in use for this very important part of the cluster creation process will enhance the speed of adding new features and removing bugs. | ||
|
|
||
| ### Speed | ||
|
|
||
| Until now, the `cloud-config-downloader` runs in a loop every `60s` to check if something changed on the shoot which requires modifications on the worker node. This produces a lot of unneeded traffic on the API server and wastes time, it will sometimes take up to `60s` until a desired modification is started on the worker node. | ||
| By writing a "real" Kubernetes controller, we can watch for the `Node`, the `OSC` in the `Secret`, and the shoot-access token in the `secret`. If any of these object changed, and only then, the required action will take effect immediately. | ||
| This will speed up operations and will reduce the load on the API server of the shoot especially for large clusters. | ||
|
|
||
| ## Scalability | ||
|
|
||
| The `cloud-config-downloader` adds a random wait time before restarting the `kubelet` in case the `kubelet` was updated or a configuration change was made to it. This is required to reduce the load on the API server and the traffic on the internet uplink. It also reduces the overall downtime of the services in the cluster because every `kubelet` restart transforms a node for several seconds into `NotReady` state which potentionally interrupts service availability. | ||
|
|
||
| Decision was made to keep the existing jitter mechanism which calculates the `kubelet-download-and-restart-delay-seconds` on the controller itself. | ||
|
|
||
| ### Correctness | ||
|
|
||
| The configuration of the `cloud-config-downloader` is actually done by placing a file for every configuration item on the disk on the worker node. This was done because parsing the content of a single file and using this as a value in `bash` reduces to something like `VALUE=$(cat /the/path/to/the/file)`. Simple, but it lacks validation, type safety and whatnot. | ||
| With the `gardener-node-agent` we introduce a new API which is then stored in the `gardener-node-agent` `secret` and stored on disk in a single YAML file for comparison with the previous known state. This brings all benefits of type safe configuration. | ||
| Because actual and previous configuration are compared, removed files and units are also removed and stopped on the worker if removed from the `OSC`. | ||
|
|
||
| ### Availability | ||
|
|
||
| Previously, the `cloud-config-downloader` simply restarted the systemd units on every change to the `OSC`, regardless which of the services changed. The `gardener-node-agent` first checks which systemd unit was changed, and will only restart these. This will prevent unneeded `kubelet` restarts. | ||
|
|
||
| ### Future Development | ||
|
|
||
| The `gardener-node-agent` opens up the possibilty for further improvements. | ||
|
|
||
| Necessary restarts of the `kubelet` could be deterministic instead of the aforementioned random jittering. In that case, the `gardenlet` could add annotations across all nodes. As the `gardener-node-agent` watches the `Node` object, it could wait with `kubelet` restarts, OSC changes or react immediately. Critical changes could be performed in chunks of nodes in serial order, but an equal time spread is possible, too. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,19 @@ | ||
| --- | ||
| apiVersion: nodeagent.config.gardener.cloud/v1alpha1 | ||
| kind: NodeAgentConfiguration | ||
| clientConnection: | ||
| qps: 100 | ||
| burst: 130 | ||
| kubeconfig: path/to/kubeconfig | ||
| logLevel: info | ||
| logFormat: text | ||
| debugging: | ||
| enableProfiling: false | ||
| enableContentionProfiling: false | ||
| featureGates: {} | ||
| operatingSystemConfigSecretName: name-of-osc-secret | ||
| accessTokenSecretName: name-of-access-token-secret | ||
| image: gardener-node-agent-image:v1 | ||
| hyperkubeImage: hyperkube-image:v2 | ||
| kubernetesVersion: 1.28.2 | ||
| # kubeletDataVolumeSize: 50Gi |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.