[eem] remove history transforms #193999

klacabane · 2024-09-25T13:18:24Z

Summary

Remove history and backfill transforms, leaving latest transform in place.

Notable changes to latest transform:

it does not read from history output anymore but source indices defined on the definition
it defines a latest.lookbackPeriod to limit the amount of data ingested, which defaults to 24h
each metadata aggregation now accepts a metadata.aggregation.lookbackPeriod which defaults to the latest.lookbackPeriod
entity.firstSeenTimestamp is removed. this should be temporary until we have a solution for https://github.com/elastic/elastic-entity-model/issues/174
latest metrics used to get the latest pre-computed value from history data, but is it now aggregating over the lookbackPeriod in the source indices (which can be filtered down with metrics.filter)
latest block on the entity definition is now mandatory

obltmachine · 2024-09-25T13:18:41Z

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

/oblt-deploy : Deploy a Kibana instance using the Observability test environments.
run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

… src/core/server/integration_tests/ci_checks'

klacabane · 2024-09-30T11:56:14Z

x-pack/packages/kbn-entities-schema/src/schema/entity_definition.ts

@@ -57,6 +52,18 @@ export const entityDefinitionSchema = z.object({
    ])
  ),
  installStartedAt: z.optional(z.string()),
+  installedComponents: z.optional(


this allows cleaning up versions of definitions that contain history components

klacabane · 2024-09-30T11:56:43Z

x-pack/plugins/entity_manager/common/constants_entities.ts

-export const ENTITY_DEFAULT_METADATA_LIMIT = 1000;
+export const ENTITY_DEFAULT_LATEST_FREQUENCY = '1m';
+export const ENTITY_DEFAULT_LATEST_SYNC_DELAY = '60s';
+export const ENTITY_DEFAULT_METADATA_LIMIT = 10;


took the opportunity to reduce the default limit

kibana-ci · 2024-10-07T12:48:51Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: 60d9ec4
Storybooks Preview
Kibana Serverless Image: docker.elastic.co/kibana-ci/kibana-serverless:pr-193999-60d9ec4b3083

Failed CI Steps

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`inventory`	222.8KB	222.7KB	-30.0B

History

💔 Build #239754 failed 6a457c4
💔 Build #239699 failed 5fc1130
💛 Build #239438 was flaky 5f458dd
💔 Build #239403 failed e379a12
💚 Build #238788 succeeded b2e80c0

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

miltonhultgren

LGTM!

miltonhultgren · 2024-10-08T14:37:25Z

x-pack/plugins/entity_manager/server/saved_objects/entity_definition.ts


 export const SO_ENTITY_DEFINITION_TYPE = 'entity-definition';

+export const backfillInstalledComponents: SavedObjectModelDataBackfillFn<


klacabane · 2024-10-09T08:57:45Z

@jaredburgettelastic @hop-dev I'll be merging that change soon, could you please verify that it doesn’t break anything that would not be covered by tests ?

hop-dev · 2024-10-09T10:35:44Z

@klacabane please go ahead and merge when you're ready, we have a big PR incoming and will fix any issues in that

elasticmachine · 2024-10-09T15:44:07Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: 02ca50e
Storybooks Preview
Kibana Serverless Image: docker.elastic.co/kibana-ci/kibana-serverless:pr-193999-02ca50e775ba

Failed CI Steps

Jest Integration Tests #4

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`inventory`	222.7KB	222.7KB	-30.0B

History

💚 Build #240690 succeeded 99ebedc
💔 Build #240638 failed 55dae11
💔 Build #240566 failed 6d189dd
💔 Build #240537 failed ef2abd2
💔 Build #240362 failed 5ddf79b

simianhacker

I added some comments about changing the entity.id generation with this change. Since this PR makes a significant change, I'd advocate for changing that too.

simianhacker · 2024-10-09T17:43:57Z

x-pack/plugins/entity_manager/server/lib/entities/ingest_pipeline/generate_latest_processors.ts

+    {
+      script: {
+        description: 'Generated the entity.id field',
+        source: cleanScript(`
+        // This function will recursively collect all the values of a HashMap of HashMaps
+        Collection collectValues(HashMap subject) {
+          Collection values = new ArrayList();
+          // Iterate through the values
+          for(Object value: subject.values()) {
+            // If the value is a HashMap, recurse
+            if (value instanceof HashMap) {
+              values.addAll(collectValues((HashMap) value));
+            } else {
+              values.add(String.valueOf(value));
+            }
+          }
+          return values;
+        }
+
+        // Create the string builder
+        StringBuilder entityId = new StringBuilder();
+
+        if (ctx["entity"]["identity"] != null) {
+          // Get the values as a collection
+          Collection values = collectValues(ctx["entity"]["identity"]);
+
+          // Convert to a list and sort
+          List sortedValues = new ArrayList(values);
+          Collections.sort(sortedValues);
+
+          // Create comma delimited string
+          for(String instanceValue: sortedValues) {
+            entityId.append(instanceValue);
+            entityId.append(":");
+          }
+
+            // Assign the entity.id
+          ctx["entity"]["id"] = entityId.length() > 0 ? entityId.substring(0, entityId.length() - 1) : "unknown";
+        }
+       `),
+      },
+    },
+    {
+      fingerprint: {
+        fields: ['entity.id'],
+        target_field: 'entity.id',
+        method: 'MurmurHash3',
+      },
+    },


@tommyers-elastic I kind of wonder if we shouldn't take this opportunity to simplify the entity.id generation? I purpose we remove the hashing AND change the entity.id to be:

{ set: { field: 'entity.id', value: definition.identityFields.map((identityField) => `{{entity.identity.${identityField.field)}}`).join(':'), } }

For something like ['service.name', 'service.environment'] it would look like {{entity.identity.service.name}}:{{entity.identity.service.environment}}

Thoughts @miltonhultgren @klacabane

This makes sense and it is implemented in #193652. I'd love to keep these separate as this PR has been opened for a while now and all the lights are finally green

➕ ➕ lets get this merged!

MERGE IT!

@klacabane Good call... ❤️

kibanamachine · 2024-10-09T21:15:57Z

Starting backport for target branches: 8.x

https://github.com/elastic/kibana/actions/runs/11263190736

### Summary Remove history and backfill transforms, leaving latest transform in place. Notable changes to latest transform: - it does not read from history output anymore but source indices defined on the definition - it defines a `latest.lookbackPeriod` to limit the amount of data ingested, which defaults to 24h - each metadata aggregation now accepts a `metadata.aggregation.lookbackPeriod` which defaults to the `latest.lookbackPeriod` - `entity.firstSeenTimestamp` is removed. this should be temporary until we have a solution for elastic/elastic-entity-model#174 - latest metrics used to get the latest pre-computed value from history data, but is it now aggregating over the `lookbackPeriod` in the source indices (which can be filtered down with `metrics.filter`) - `latest` block on the entity definition is now mandatory --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Mark Hopkin <mark.hopkin@elastic.co> (cherry picked from commit 8f8e988)

kibanamachine · 2024-10-09T21:20:27Z

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

# Backport This will backport the following commits from `main` to `8.x`: - [[eem] remove history transforms (#193999)](#193999)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport)  Co-authored-by: Kevin Lacabane <kevin.lacabane@elastic.co>

remove history transform

7359805

kibanamachine and others added 18 commits September 25, 2024 13:30

[CI] Auto-commit changed files from 'node scripts/yarn_deduplicate'

46423e5

[CI] Auto-commit changed files from 'node scripts/jest_integration -u…

be50f9d

… src/core/server/integration_tests/ci_checks'

fix tests

87f2632

Merge branch 'main' into 173-remove-history-transforms

c1cf4c7

fix aggs

f01a8a0

latest lookbackPeriod

8d0df93

remove ccs index

eef049e

fix tests

238a3c9

fix tests

1d9441e

fix security tests

6cb7e26

update validate_transform_ids

9f43b89

remove deprecated files

32f33f7

fix tests

97c0270

fix tests

7ea3236

fix test

eb3a923

fix test

c2a163d

update snapshots

31cd18a

Merge branch 'main' into 173-remove-history-transforms

3f28a24

klacabane commented Sep 30, 2024

View reviewed changes

klacabane marked this pull request as ready for review September 30, 2024 12:23

klacabane requested review from a team as code owners September 30, 2024 12:23

klacabane requested a review from machadoum September 30, 2024 12:23

Merge branch 'main' into 173-remove-history-transforms

ad391ae

klacabane added release_note:skip Skip the PR/issue when compiling release notes Feature:EEM Elastic Entity Model labels Sep 30, 2024

remove firstSeenTimestamp investigate_app

6a457c4

klacabane requested a review from a team as a code owner October 7, 2024 09:54

botelastic bot added the ci:project-deploy-observability Create an Observability project label Oct 7, 2024

Merge branch 'main' into 173-remove-history-transforms

60d9ec4

Merge branch 'main' into 173-remove-history-transforms

f2afe29

miltonhultgren approved these changes Oct 8, 2024

View reviewed changes

hop-dev and others added 4 commits October 8, 2024 17:11

Merge branch 'main' into 173-remove-history-transforms

5ddf79b

fix entity analytics tests

ef2abd2

Merge branch 'main' into 173-remove-history-transforms

73f0a7d

remove interval option

6d189dd

klacabane added 3 commits October 9, 2024 13:10

Merge branch 'main' into 173-remove-history-transforms

55dae11

Merge branch 'main' into 173-remove-history-transforms

99ebedc

Merge branch 'main' into 173-remove-history-transforms

02ca50e

simianhacker reviewed Oct 9, 2024

View reviewed changes

jaredburgettelastic approved these changes Oct 9, 2024

View reviewed changes

hop-dev requested review from simianhacker and removed request for machadoum October 9, 2024 20:47

simianhacker approved these changes Oct 9, 2024

View reviewed changes

simianhacker merged commit 8f8e988 into elastic:main Oct 9, 2024
51 checks passed

kibanamachine added the v9.0.0 label Oct 9, 2024

kibanamachine mentioned this pull request Oct 9, 2024

[8.x] [eem] remove history transforms (#193999) #195680

Merged

kibanamachine added the v8.16.0 label Oct 9, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[eem] remove history transforms #193999

[eem] remove history transforms #193999

klacabane commented Sep 25, 2024 •

edited by kibanamachine

Loading

obltmachine commented Sep 25, 2024

klacabane Sep 30, 2024 •

edited

Loading

klacabane Sep 30, 2024

kibana-ci commented Oct 7, 2024 •

edited

Loading

miltonhultgren left a comment

miltonhultgren Oct 8, 2024

klacabane commented Oct 9, 2024

hop-dev commented Oct 9, 2024

elasticmachine commented Oct 9, 2024 •

edited

Loading

simianhacker left a comment

simianhacker Oct 9, 2024

klacabane Oct 9, 2024

hop-dev Oct 9, 2024

simianhacker Oct 9, 2024 •

edited

Loading

kibanamachine commented Oct 9, 2024

kibanamachine commented Oct 9, 2024


		export const SO_ENTITY_DEFINITION_TYPE = 'entity-definition';

		export const backfillInstalledComponents: SavedObjectModelDataBackfillFn<

[eem] remove history transforms #193999

[eem] remove history transforms #193999

Conversation

klacabane commented Sep 25, 2024 • edited by kibanamachine Loading

Summary

obltmachine commented Sep 25, 2024

🤖 GitHub comments

klacabane Sep 30, 2024 • edited Loading

Choose a reason for hiding this comment

klacabane Sep 30, 2024

Choose a reason for hiding this comment

kibana-ci commented Oct 7, 2024 • edited Loading

💛 Build succeeded, but was flaky

Failed CI Steps

Metrics [docs]

Async chunks

History

miltonhultgren left a comment

Choose a reason for hiding this comment

miltonhultgren Oct 8, 2024

Choose a reason for hiding this comment

klacabane commented Oct 9, 2024

hop-dev commented Oct 9, 2024

elasticmachine commented Oct 9, 2024 • edited Loading

💛 Build succeeded, but was flaky

Failed CI Steps

Metrics [docs]

Async chunks

History

simianhacker left a comment

Choose a reason for hiding this comment

simianhacker Oct 9, 2024

Choose a reason for hiding this comment

klacabane Oct 9, 2024

Choose a reason for hiding this comment

hop-dev Oct 9, 2024

Choose a reason for hiding this comment

simianhacker Oct 9, 2024 • edited Loading

Choose a reason for hiding this comment

kibanamachine commented Oct 9, 2024

kibanamachine commented Oct 9, 2024

💚 All backports created successfully

Questions ?

klacabane commented Sep 25, 2024 •

edited by kibanamachine

Loading

klacabane Sep 30, 2024 •

edited

Loading

kibana-ci commented Oct 7, 2024 •

edited

Loading

elasticmachine commented Oct 9, 2024 •

edited

Loading

simianhacker Oct 9, 2024 •

edited

Loading