[EEM] Remove duplicates from latest data set

[miltonhultgren]

By only grouping on entity.id we should be able to remove duplicates in the latest indices.
This PR also removes the values found for entity.identityFields and replaces it with a list of those field names.
This PR also lifts the values for the identity fields to the root of the document.
This PR removes the displayName from the historical documents.

How to test

Source data:

PUT index_a
{
  "mappings": {
    "properties": {
      "a": {
        "type": "keyword"
      },
      "@timestamp": {
        "type": "date"
      }
    }
  }
}

PUT index_b
{
  "mappings": {
    "properties": {
      "b": {
        "type": "keyword"
      },
      "@timestamp": {
        "type": "date"
      }
    }
  }
}

POST index_a/_doc
{
  "a": "same",
  "@timestamp": "2024-07-05T12:33:06.162Z"
}

POST index_b/_doc
{
  "b": "same",
  "@timestamp": "2024-07-05T12:33:06.162Z"
}

Entity definition:

POST kbn:/internal/api/entities/definition
{
  "id": "bucket_key",
  "name": "Bucket key",
  "type": "service",
  "indexPatterns": [
    "index_*"
  ],
  "timestampField": "@timestamp",
  "lookback": "5m",
  "identityFields": [
    {
      "field": "a",
      "optional": true
    },
    {
      "field": "b",
      "optional": true
    }
  ],
  "displayNameTemplate": "{{a}}{{b}}",
  "history": {
    "timestampField": "@timestamp",
    "interval": "5m"
  }
}

Change in the format of the resulting documents

"identityFields": {
  "a": null,
  "b": "same"
},

=>

"identityFields": [
  "a",
  "b"
],

[obltmachine]

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

• /oblt-deploy : Deploy a Kibana instance using the Observability test environments.
• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

[tommyers-elastic]

this is the right direction

can we also remove displayName from the history docs (since it should only use one (and maybe not the one in the current doc) identityField), and add it to the latest docs

[miltonhultgren]

I'm going to split this PR in two:

1. Lifting identifyFields up to root in both datasets
2. Removing duplication in the latest dataset with tests that use DataForge for data generation

[miltonhultgren]

can we also remove displayName from the history docs (since it should only use one (and maybe not the one in the current doc) identityField), and add it to the latest docs

@tommyers-elastic
So, in the history docs we would track the values of the identity fields (by storing them at the root).
And then we'd grab those values in the latest transform, and expect to find a single value and use that value to create the display name?

Are we sure we won't need the display name for the history documents? In the UI I guess not because we'll likely enter the history from the latest, so we can hang on to the display name from there.

But what about if there are more than one value found in the latest transform, I'm still not sure how to handle that.
I almost feel like it would be correct to have the ingest pipeline throw errors for that.
What do you think? (In either case, I'll likely address this in the next PR)

Also, this PR is ready for review again!

[tommyers-elastic]

@miltonhultgren the reason for displayName in latest only is because of:

say you have an entity definition with two fields that both contain the user identifier, user.name and labels.user_name; in the history documents you have one or the other. so you might construct a template like {{#user.name}}{{.}}{{/user.name}}{{#labels.user_name}}{{.}}{{/labels.user_name}}. but then when these are combined in the latests docs and both identity fields exist in the document, you get a display name with both fields populated.

so you just put a display name with one of the fields like {{user.name}}. if you include display name in the history documents, only some of them get a value populated, but now in the latest transform you get the correct value.

[tommyers-elastic]

LGTM - but do we need to also update the component templates?

the changes i think we need are to remove displayName from the base template, and include it only in the latest; and to map identityFields as a keyword in the base template.

(while we're at it, we should remove firstSeenTimestamp from the shared mapping too, and include that alongside displayName in the latest template only)

[tommyers-elastic]

👍

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: be52863
• Storybooks Preview
• Kibana Serverless Image: docker.elastic.co/kibana-ci/kibana-serverless:pr-187699-be528630ad3e
• Observability Deployment

Metrics [docs]

✅ unchanged

History

• 💚 Build #220942 succeeded ea9a00b
• 💛 Build #220888 was flaky 34a463f
• 💔 Build #220415 failed 54b70e1

cc @miltonhultgren

[simianhacker]

If it's a single level field like tags, you don't need to initialize the new HashMap();

When you refactored this code you missed where it tests the currentIndex + 1 === parts.length. If parts.length is 1 and the currentIndex is 0 then you're already at the end and you can just assign the value, there is no need to instantiate a new HashMap().

[simianhacker]

You're at the end of the path, there is no need for this.

[simianhacker]

Since these are single value fields, I wonder if we shouldn't have used top_metric and then used the same set processor as the history?