[EEM] Include identity fields and values in document root

[miltonhultgren]

Goal of this PR:
For a definition that sets host.name as an identity field, the history document produced should have host.name set at the root of the document to the value found.

For the latest documents, we should collect all the found values for host.name and set it to the root as well.

Maybe remove displayName from the historical documents.

Undecided how we handle the case where multiple values are found in the latest ingest pipeline.
See #187699 (comment) and #187699 (comment)

[obltmachine]

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

• /oblt-deploy : Deploy a Kibana instance using the Observability test environments.
• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

[miltonhultgren]

is it necessary to have this complex logic vs something more simple like

    ...definition.identityFields.map((key) => {
      return {
        set: {
          field: key.field,
          value: `{{entity.identityFields.${key.field}}}`,
          if: `ctx.entity?.identityFields?.${key.field.replaceAll('.', '?.')} != null`,
        }
      }
    }),

does this correctly handle the nested object behaviour?

[tommyers-elastic]

did this get tested? i think it should work?

[miltonhultgren]

Yes, it works! Put it back into the original PR!

[miltonhultgren]

Merged this work back into #187699 since it made it easier after all.