Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ML] AIOps: Supports text fields in log rate analysis #165124

Merged
merged 58 commits into from
Oct 4, 2023
Merged

[ML] AIOps: Supports text fields in log rate analysis #165124

merged 58 commits into from
Oct 4, 2023

Conversation

walterra
Copy link
Contributor

@walterra walterra commented Aug 29, 2023 •
edited
Loading

Summary

Part of #167467.

Adds support for text fields in log pattern analysis. Text fields will be analysed using log categorization, similar to log pattern analysis. Significant log patterns will be identified using the chi2test package, similar to how we detect data drifts.

image

Checklist

@walterra walterra self-assigned this Aug 29, 2023
@walterra walterra force-pushed the 163690-ml-aiops-log-rate-text-field-support branch 2 times, most recently from 37f24df to 824ca30 Compare August 30, 2023 07:29
@walterra walterra mentioned this pull request Sep 1, 2023
Closed
@walterra walterra force-pushed the 163690-ml-aiops-log-rate-text-field-support branch 5 times, most recently from 7400379 to 2b8953e Compare September 27, 2023 08:47
@walterra walterra force-pushed the 163690-ml-aiops-log-rate-text-field-support branch from 2b8953e to 7fdce57 Compare September 27, 2023 17:03
@walterra walterra mentioned this pull request Sep 28, 2023
Closed
@walterra walterra force-pushed the 163690-ml-aiops-log-rate-text-field-support branch from 7fdce57 to df03ec3 Compare September 28, 2023 12:09
@walterra
Copy link
Contributor Author

walterra commented Oct 3, 2023

@jgowdyelastic I added this icon+tooltip message now for text fields for this PR to have at least some clarification that this is a log pattern. I'd like to be able to improve this in the future and give users the option to show more examples.

image

peteharverson
peteharverson approved these changes Oct 3, 2023
View reviewed changes
Copy link
Contributor

@peteharverson peteharverson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Latest changes LGTM

Tested styling used for text fields in the table as shown in #165124 (comment) and looks good.

peteharverson
peteharverson reviewed Oct 3, 2023
View reviewed changes
Copy link
Contributor

@peteharverson peteharverson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As discussed, just hit an error with row expansion with the grouped table.

@peteharverson peteharverson self-requested a review October 3, 2023 17:05
@walterra
Copy link
Contributor Author

walterra commented Oct 3, 2023

@peteharverson I was able to fix the render loop with multiple expanded rows in 6587927.

peteharverson
peteharverson approved these changes Oct 3, 2023
View reviewed changes
Copy link
Contributor

@peteharverson peteharverson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested latest changes to fix render loop and LGTM

jgowdyelastic
jgowdyelastic approved these changes Oct 3, 2023
View reviewed changes
Copy link
Member

@jgowdyelastic jgowdyelastic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Latest changes LGTM

@walterra walterra enabled auto-merge (squash) October 3, 2023 23:11
@walterra walterra disabled auto-merge October 4, 2023 05:29
@kibana-ci
Copy link
Collaborator

💚 Build Succeeded

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id before after diff
aiops 480 481 +1
dataVisualizer 578 579 +1
ml 1817 1818 +1
transform 387 388 +1
total +4

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
aiops 595.8KB 586.5KB -9.3KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id before after diff
aiops 8.0KB 8.0KB +28.0B
Unknown metric groups

API count

id before after diff
@kbn/ml-agg-utils 93 99 +6

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @walterra

@walterra walterra merged commit d8886d8 into elastic:main Oct 4, 2023
@kibanamachine kibanamachine added the backport:skip This commit does not require backporting label Oct 4, 2023
@walterra walterra mentioned this pull request Oct 5, 2023
Merged
2 tasks
@walterra walterra deleted the 163690-ml-aiops-log-rate-text-field-support branch October 5, 2023 08:32
@walterra walterra mentioned this pull request Oct 11, 2023
Closed
@szabosteve szabosteve changed the title [ML] AIOps: Support text fields in log rate analysis [ML] AIOps: Supports text fields in log rate analysis Oct 12, 2023
@peteharverson peteharverson mentioned this pull request Dec 8, 2023
Closed
@walterra walterra mentioned this pull request Apr 18, 2024
Closed
@walterra walterra mentioned this pull request Jul 5, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jgowdyelastic jgowdyelastic jgowdyelastic approved these changes

@peteharverson peteharverson peteharverson approved these changes

@alvarezmelissa87 alvarezmelissa87 Awaiting requested review from alvarezmelissa87

Assignees

@walterra walterra

Labels
backport:skip This commit does not require backporting Feature:ML/AIOps ML AIOps features: Change Point Detection, Log Pattern Analysis, Log Rate Analysis :ml release_note:enhancement v8.11.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@walterra @elasticmachine @peteharverson @jgowdyelastic @kibana-ci @kibanamachine