Disable merge JSON log in 4.0

[jcantrill]

Issue

Reviewing logs from a production cluster led to the creation of [1]. During discussion in the issue and slack, it its my understanding the root cause is:

• messages that are JSON logged are merged into the fluent payload by default
• Elasticsearch dynamic mappings cause a new mapping to be generated when a field is first indexed
• Subsequent indexing will fail if the value in the later documents does not match the original (e.g. int vs. string)

This subsequently is leading to message loss as identified in [1] since the log entry is rejected by ES (Does it cycle indefinitely?).

Proposal

This proposal would disable the merge JSON log feature by default in the 4.0 delivery and would:

• Make it difficult to revert (operator=unmanaged, set env var?)
• Document deprecation, ramifications if enabled
• Introduce a JIRA card to re-enable in an alternate capacity (e.g. whitelist, project logging CR, subfield)

[1] Ref: #1491

[richm]

NACK - many customers use and depend on this feature

[nhosoi]

It may not affect us, but merge_json_log is discontinued from fluent-plugin-kubernetes_metadata_filter:
NOTE: As of the release 2.1.x of this plugin, it no longer supports parsing the source message into JSON and attaching it to the payload. The following configuration options are removed:

• merge_json_log
• preserve_json_log

[jcantrill]

@richm We should consider this in context of the proposed release, 4.0. The release is green-field only deployments. In the current incarnation, this feature is broken:

• It stomps existing fields
• It shows we have data loss, possibly a memory leak of sorts where records cycle back to retry queues
• It does not properly handle the case where 2 containers create log messages with the same fields but of different types

This proposal does not advocate removing it, but disabling it by default and adding documentation to clearly define the drawbacks. Until we can devise a proper solution, customer's should understand what they are getting and why we have disabled it.

[richm]

It may not affect us, but merge_json_log is discontinued from fluent-plugin-kubernetes_metadata_filter:
NOTE: As of the release 2.1.x of this plugin, it no longer supports parsing the source message into JSON and attaching it to the payload. The following configuration options are removed:

* `merge_json_log`

* `preserve_json_log`

Right - it doesn't affect us (it is us who removed that feature from fluent-plugin-kubernetes_metadata_filter . . . because of this problem) - we have our own implementation - https://github.com/openshift/origin-aggregated-logging/blob/master/fluentd/configs.d/filter-parse-json-field.conf

[richm]

ok - so we disable it by default, and only for 4.0 - to your list of tasks I would add

• change the json-parsing test to enable the feature before testing (by setting cluster to unmanaged, etc.)
• come up with a real solution to the problem in 4.1

[jcantrill]

• come up with a real solution to the problem in 4.1
  https://jira.coreos.com/browse/LOG-314

[jcantrill]

Partially addressed in openshift/cluster-logging-operator#74