[observability] node-exporter insufficient capabilities

[geropl]

I see this error when deploying with-observability: https://werft.gitpod-dev.com/job/gitpod-build-gpl-tracing-disable-errors.7#deploy:deploy

Error: kubectl rollout status -n staging-gpl-tracing-disable-errors daemonset node-exporter exit with non-zero status code

kubectl describe ds node-exporter shows:

Warning  FailedCreate  13s (x15 over 31m)  daemonset-controller  Error creating: pods "node-exporter-" is forbidden: PodSecurityPolicy: unable to admit pod: [spec.containers[0].securityContext.capabilities.add: Invalid value: "CAP_SYS_TIME": capability may not be added]

The PSP is there, but not namespaced (so overriden/re-used by every monitoring-satellite installation) and has

RequiredDropCapabilities:
- All

configured.

[ArthurSens]

I believe this is an issue from upstream. We're making kube-prometheus adopt security best practices, like dropping unnecessary capabilities, but looks like we forgot to update the pod-security policies

[fullmetalrooster]

Here is the PR that should fix the problem. prometheus-operator/kube-prometheus#1642

[ArthurSens]

The fix should have been imported to previews here: gitpod-io/observability#88

I'll deploy in a new env just to test

[ArthurSens]

Confirmed that it is running!