[BUG] Including hostname in json logs drops kube_cluster_name tag

[ryanartecona]

Agent Environment

> agent version
Agent 7.39.2 - Commit: f6f7bc9 - Serialization version: v5.0.26 - Go version: go1.17.11

Running the agent as a k8s daemonset via the helm chart using mostly default values. Logs are enabled with container_collect_all: true.

Describe what happened:

I have an app printing json logs that are collected by the agent. If a log line includes hostname as a top level key in the json object, that value will override what datadog shows in the Host field for that line, causing it to lose association with the server host the log came from. Also, kube_cluster_name will not be added as a tag to that log line. This is a similar problem to the one in #14099.

Describe what you expected:

Normal log lines behave as expected. Specifically, if a log line does not include hostname in the json, datadog will show it as coming from the correct server and will show kube_cluster_name tag which is automatically added by the agent.

Steps to reproduce the issue:

1. Run datadog-agent as a daemonset in k8s with logs enabled, a cluster name set, and container_collect_all: true.
2. Start an app pod whose container runs the following bash script:

   while true; do 
     echo '{"message":"this line works properly"}'; 
     echo '{"message":"this line is buggy","hostname":"nonsense"}'; 
     sleep 10; 
   done;
   

Additional environment details (Operating System, Cloud provider, etc):

[vickenty]

Hi @ryanartecona, thanks for using Datadog.

This issue queue is primarily intended for tracking features, bugs and work items associated with the datadog-agent open source project. For your request, please contact our support.

[nicolas-vivot]

I confirm this bug. It drops all tags added by the kubernetes datadog agent when the hostname attribute is present inside json formatted logs.

@ryanartecona did you get any reply from the support ?

[nicolas-vivot]

After contacting the support, here is the explanation.

As described here, by default hostname is considered by the json preprocessing configuration to remap the host name.

Therefore the hostname attribute won't have the same value as expected. And this has the consequence of dropping all host related tags because tags are coupled with the hostname, as further explained by the support (screenshot of the complete answer below)

(tags are associated to hostname and not sent with logs every time for performance reason, therefore this drop effect happens when the hostname attribute is overriden on logs with a value that does not match the real host where Datadog agents are running on.

How to resolve that: get rid of the hostname attribute coming from your json logs, or pass the real host value to a HOSTNAME environment variable to your container to get the correct one.

Support answer: