missing pod metrics on namespaces with pod-security.kubernetes.io/enforce label

[danfinn]

Recently we upgraded from 8.14.x to 8.15.3 to fix a bug related to service account tokens. It seems as though there is a change in 8.15.3 where some pod metrics are not being collected from namespaces that have the pod-security.kubernetes.io/enforce label. Specifically we noticed that the kubernetes.pod.cpu.usage.limit.pct metrics are missing from these namespaces however there might be others.

Here is an example describe on a namespace that is missing the cpu usage limit pct metrics:

Name:         namespace_name
Labels:       bmap-control-plane=true
              kubernetes.io/metadata.name=namespace_name
              pod-security.kubernetes.io/enforce=baseline
              purpose=namespace_name
Annotations:  <none>
Status:       Active

No resource quota.

No LimitRange resource.

and the only difference I can find between all the other namespaces that are still acting normally is the pod-security label.

If I remove this label metricbeat will start to provide the missing metric. Adding the label back causes the metrics to no longer be shipped.

some examples from grafana of working vs not:

[botelastic]

This issue doesn't have a Team:<team> label.

[danfinn]

I did try with the latest 8.16.3 docker image and it did not fix the issue.