container_fs metrics reporting root device metrics with containerd

[sidewinder12s]

I've deployed cAdvisor with the v0.45.0-containerd-cri tag and used the reference deploy manifests along with EKS 1.23 and containerd (using the AWS EKS AMI).

Any idea why all the contianer_fs_* metrics would be reporting the metrics of the root device and not each container? I do get series with correct labels per container, its just that all of the metrics are reporting the root disks free space, used space, etc.

DS Spec for reference:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    app.kubernetes.io/instance: cadvisor
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: cadvisor
    app.kubernetes.io/version: v0.45.0-containerd-cri
    helm.sh/chart: cadvisor-0.1.1
  name: cadvisor
  namespace: cadvisor
spec:
  selector:
    matchLabels:
      app.kubernetes.io/instance: cadvisor
      app.kubernetes.io/name: cadvisor
  template:
    metadata:
      labels:
        app.kubernetes.io/instance: cadvisor
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/name: cadvisor
        app.kubernetes.io/version: v0.45.0-containerd-cri
        helm.sh/chart: cadvisor-0.1.1
    spec:
      automountServiceAccountToken: false
      containers:
        - args:
            - '--housekeeping_interval=10s'
            - '--max_housekeeping_interval=15s'
            - '--event_storage_event_limit=default=0'
            - '--event_storage_age_limit=default=0'
            - '--enable_metrics=app,cpu,disk,diskIO,memory,network,process'
            - '--docker_only'
            - '--store_container_labels=false'
            - >-
              --whitelisted_container_labels=io.kubernetes.container.name,io.kubernetes.pod.name,io.kubernetes.pod.namespace
          image: >-
            cadvisor:v0.45.0-containerd-cri
          imagePullPolicy: IfNotPresent
          name: cadvisor
          ports:
            - containerPort: 8080
              name: http
              protocol: TCP
          resources:
            requests:
              cpu: 25m
              memory: 50Mi
          securityContext: {}
          volumeMounts:
            - mountPath: /rootfs
              name: rootfs
              readOnly: true
            - mountPath: /var/run
              name: var-run
              readOnly: true
            - mountPath: /sys
              name: sys
              readOnly: true
            - mountPath: /var/lib/docker
              name: docker
              readOnly: true
            - mountPath: /dev/disk
              name: disk
              readOnly: true
      priorityClassName: system-node-critical
      serviceAccountName: cadvisor
      terminationGracePeriodSeconds: 30
      tolerations:
        - operator: Exists
      volumes:
        - hostPath:
            path: /
          name: rootfs
        - hostPath:
            path: /var/run
          name: var-run
        - hostPath:
            path: /sys
          name: sys
        - hostPath:
            path: /var/lib/docker
          name: docker
        - hostPath:
            path: /dev/disk
          name: disk

Related: #2785

[jlaino]

Any update on this issue? sidewinder12s Have you been able to solve this problem?

[sidewinder12s]

Nope, gave up on it and moved to a summary exporter.