OTAGENT-286 Add support for filelog receiver

[krlv]

What this PR does / why we need it:

To collect logs in Kubernetes environments, the filelog receiver requires specific volumes to be mounted to the otel-agent container:

• /var/log/pods
• /var/log/containers
• /var/lib/docker/containers

Currently, the Helm chart does not provide a way to mount these additional volumes for the otel-agent container.

Proposed Solution

Introduce new Helm chart parameters to enable filelog receiver support and allow mounting additional volumes in the otel-agent container:

• datadog.otelCollector.logs.enabled:
  When set to true, the Helm chart will check the OTel Collector configuration for a filelog receiver definition. If the receiver is defined, it mounts the required volumes to collect logs from containers and pods.

• agents.containers.otelAgent.volumeMounts:
  Specify additional volume mounts in the otel-agent container.

Example configuration:

datadog:
  otelCollector:
    enabled: false
    ## Provide OTel Collector logs configuration
    logs:
      # datadog.otelCollector.logs.enabled -- When true, Helm checks the OTel Collector configuration for a `filelog` receiver.
      # If a `filelog` receiver is defined, it mounts the required volumes to collect logs from containers and pods.
      enabled: false

agents:
  containers:
    otelAgent:
      # agents.containers.otelAgent.volumeMounts -- Specify additional volumes to mount in the otel-agent container.
      volumeMounts: []
      #   - name: <VOLUME_NAME>
      #     mountPath: <CONTAINER_PATH>
      #     readOnly: true

This implementation maintains backward compatibility by requiring explicit configuration changes to enable the new behavior.

Comparison with the OpenTelemetry Helm Chart and Operator

The OpenTelemetry Collector Helm Chart allows customers to configure the filelog receiver in two ways:

Logs Collection Preset

When enabled, the chart adds a filelogreceiver to the logs pipeline. This receiver is configured to read the files where the Kubernetes container runtime writes all containers' console output (e.g. /var/log/pods/*/*/*.log).

Reference Links:

• Filelog Receiver Component
• Components for Kubernetes: Filelog Receiver
• Logs Collection Preset

Manual Configuration

Customers can also manually add additional volumes and mounts using extraVolumes and extraVolumeMounts. Note that this behavior is not documented in the Helm chart docs or the GitHub README.

Reference Link:

• Helm values file

Proposed Solution Parallels

Our new parameter datadog.otelCollector.logs.enabled offers a developer experience similar to the Logs Collection Preset (presets.logsCollection.enabled) in the OpenTelemetry Collector Helm Chart. This approach should be intuitive for customers already familiar with configuring the OTel Collector via its official Helm chart. The key difference is that our solution does not implicitly change or update the logs pipeline, which avoids potential confusion from silent configuration changes. Additionally, we clearly document the use of the agents.containers.otelAgent.volumeMounts parameter.

Use Cases

1. Backward Compatibility - No Additional Volumes Mounted

In this scenario, the filelog receiver is defined in the OTel Collector configuration, but since datadog.otelCollector.logs.enabled is not explicitly set (defaults to false), no additional volumes are mounted in the otel-agent container.

This demonstrates backward compatibility by requiring explicit configuration changes to enable the new behavior.

2. Enabling Filelog Receiver Support

The customer wants to collect Kubernetes logs using the filelog receiver. By explicitly setting datadog.otelCollector.logs.enabled to true, additional volumes for paths /var/log/pods, /var/log/containers, and /var/lib/docker/containers are mounted in the otel-agent container.

DaemonSet Diff:

# Source: datadog/templates/daemonset.yaml
kind: DaemonSet
spec:
  template:
    spec:
      containers:
        - name: otel-agent
          ...
          volumeMounts:
            ...
+          - name: logpodpath
+             mountPath: /var/log/pods
+             mountPropagation: None
+             readOnly: true
+           - name: logscontainerspath
+             mountPath: /var/log/containers
+             mountPropagation: None
+             readOnly: true
+           - name: logdockercontainerpath
+             mountPath: /var/lib/docker/containers
+             mountPropagation: None
+             readOnly: true
            ...
      volumes:
        ...
+       - hostPath:
+           path: /var/lib/datadog-agent/logs
+         name: pointerdir
+       - hostPath:
+           path: /var/log/pods
+         name: logpodpath
+       - hostPath:
+           path: /var/log/containers
+         name: logscontainerspath
+       - hostPath:
+           path: /var/lib/docker/containers
+         name: logdockercontainerpath
      ...

3. Collecting Kubernetes Logs and Custom Logs

The customer wants to collect not only Kubernetes logs via the filelog receiver, but also logs from a custom location (eg. /var/log/custom). Here, datadog.otelCollector.logs.enabled is set to true, and an additional volume for the custom location is defined in agents.volumes and containers.otelAgent.volumeMounts.

The result is that standard volumes for Kubernetes logs, along with the custom volume, are mounted in the otel-agent container.

DaemonSet Diff:

# Source: datadog/templates/daemonset.yaml
apiVersion: apps/v1
kind: DaemonSet
spec:
  template:
    spec:
      containers:
        - name: otel-agent
          ...
          volumeMounts:
            ...
+           - name: logpodpath
+             mountPath: /var/log/pods
+             mountPropagation: None
+             readOnly: true
+           - name: logscontainerspath
+             mountPath: /var/log/containers
+             mountPropagation: None
+             readOnly: true
+           - name: logdockercontainerpath
+             mountPath: /var/lib/docker/containers
+             mountPropagation: None
+             readOnly: true
            - name: runtimesocketdir
              mountPath: /host/var/run
              mountPropagation: None
              readOnly: true
+           - mountPath: /var/log/custom
+             name: logscustompath
+             readOnly: true
      volumes:
        ...
+       - hostPath:
+           path: /var/lib/datadog-agent/logs
+         name: pointerdir
+       - hostPath:
+           path: /var/log/pods
+         name: logpodpath
+       - hostPath:
+           path: /var/log/containers
+         name: logscontainerspath
+       - hostPath:
+           path: /var/lib/docker/containers
+         name: logdockercontainerpath
        - name: otelconfig
          configMap:
            name: otel-agent-datadog-otel-config
            items:
              - key: otel-config.yaml
                path: otel-config.yaml
+       - hostPath:
+           path: /var/log/custom
+         name: logscustompath
      ...

4. Collecting Kubernetes and Custom Logs with Core Agent Logs Collection

In this scenario, the customer is using the Core Agent for logs collection (datadog.logs.enabled: true), along with enabled filelog receiver (datadog.otelCollector.logs.enabled: true). An additional custom volume is defined in agents.volumes and `containers.otelAgent.volumeMounts.

As a result, standard volumes for Kubernetes logs are mounted in both the core agent container (existing behavior) and the otel-agent container, while the custom volume is mounted only in the otel-agent container.

DaemonSet Diff:

# Source: datadog/templates/daemonset.yaml
apiVersion: apps/v1
kind: DaemonSet
spec:
  template:
    spec:
      containers:
        - name: agent
          ...
          volumeMounts:
            # added because of the datadog.logs.enabled: true
            - name: logpodpath
              mountPath: /var/log/pods
              mountPropagation: None
              readOnly: true
            - name: logscontainerspath
              mountPath: /var/log/containers
              mountPropagation: None
              readOnly: true
            - name: logdockercontainerpath
              mountPath: /var/lib/docker/containers
              mountPropagation: None
              readOnly: true
            - mountPath: /var/log/custom
              name: logscustompath
              readOnly: true
        - name: otel-agent
          ...
          volumeMounts:
            ...
+           - name: logpodpath
+             mountPath: /var/log/pods
+             mountPropagation: None
+             readOnly: true
+           - name: logscontainerspath
+             mountPath: /var/log/containers
+             mountPropagation: None
+             readOnly: true
+           - name: logdockercontainerpath
+             mountPath: /var/lib/docker/containers
+             mountPropagation: None
+             readOnly: true
+           - mountPath: /var/log/custom
+             name: logscustompath
+             readOnly: true
      volumes:
        # added because of the datadog.logs.enabled: true
        - hostPath:
            path: /var/lib/datadog-agent/logs
          name: pointerdir
        - hostPath:
            path: /var/log/pods
          name: logpodpath
        - hostPath:
            path: /var/log/containers
          name: logscontainerspath
        - hostPath:
            path: /var/lib/docker/containers
          name: logdockercontainerpath
        ...
+       - hostPath:
+           path: /var/log/custom
+         name: logscustompath
      ...

5. Core Agent Logs Collection with Additional Custom Volume for OTel Component

The customer uses the Core Agent for logs collection (datadog.logs.enabled: true) and requires an additional volume for a custom OTel component. Here, the filelog receiver is not defined and datadog.otelCollector.logs.enabled is set to false, so standard volumes for Kubernetes logs are mounted only in the core agent container. The additional custom volume is defined for the otel-agent container only.

DaemonSet Diff:

# Source: datadog/templates/daemonset.yaml
apiVersion: apps/v1
kind: DaemonSet
spec:
  template:
    spec:
      containers:
        - name: agent
          ...
          volumeMounts:
            # added because of the datadog.logs.enabled: true
            - name: logpodpath
              mountPath: /var/log/pods
              mountPropagation: None
              readOnly: true
            - name: logscontainerspath
              mountPath: /var/log/containers
              mountPropagation: None
              readOnly: true
            - name: logdockercontainerpath
              mountPath: /var/lib/docker/containers
              mountPropagation: None
              readOnly: true
            - mountPath: /var/log/custom
              name: logscustompath
              readOnly: true
        - name: otel-agent
          ...
          volumeMounts:
            ...
+           - mountPath: /var/log/custom
+             name: logscustompath
+             readOnly: true
      volumes:
        # added because of the datadog.logs.enabled: true
        - hostPath:
            path: /var/lib/datadog-agent/logs
          name: pointerdir
        - hostPath:
            path: /var/log/pods
          name: logpodpath
        - hostPath:
            path: /var/log/containers
          name: logscontainerspath
        - hostPath:
            path: /var/lib/docker/containers
          name: logdockercontainerpath
        ...
+       - hostPath:
+           path: /var/log/custom
+         name: logscustompath
      ...

Which issue this PR fixes

OTAGENT-286

Checklist

[Place an '[x]' (no spaces) in all applicable fields. Please remove unrelated fields.]

• Chart Version bumped
• Documentation has been updated with helm-docs (run: .github/helm-docs.sh)
• CHANGELOG.md has been updated
• Variables are documented in the README.md

[truthbk]

I see nothing wrong this.