Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ML] AIOps: Improve cleanup of default queries. #176534

Merged
merged 14 commits into from
Mar 8, 2024

Conversation

walterra
Copy link
Contributor

@walterra walterra commented Feb 8, 2024

Summary

  • Moves some query utils from the transform plugin to @kbn/ml-query-utils to make them available for the aiops plugin. This allows us to better clean up default queries before sending them off to API endpoints. Some more unit tests have been added as well as query utils to clean up the default queries we get from EUI search query bars.
  • Adds assertions for url state to aiops functional tests. These ensure that the overall time frame and window parameters for analysis get correctly set.

Checklist

@walterra walterra added the release_note:skip Skip the PR/issue when compiling release notes label Feb 8, 2024
@walterra walterra self-assigned this Feb 8, 2024
@walterra walterra force-pushed the ml-176387-fix-functional-tests branch from eafbf00 to ba76eb8 Compare February 20, 2024 13:50
@walterra walterra added :ml Feature:ML/AIOps ML AIOps features: Change Point Detection, Log Pattern Analysis, Log Rate Analysis v8.14.0 labels Feb 20, 2024
@walterra walterra marked this pull request as ready for review February 20, 2024 13:57
@walterra walterra requested a review from a team as a code owner February 20, 2024 13:57
@elasticmachine
Copy link
Contributor

Pinging @elastic/ml-ui (:ml)

@walterra walterra requested review from darnautov and qn895 February 20, 2024 13:57
@walterra walterra changed the title [ML] AIOps: Fix functional tests [ML] AIOps: Improve cleanup of default queries. Feb 20, 2024
@walterra
Copy link
Contributor Author

walterra commented Feb 20, 2024

@walterra walterra mentioned this pull request Feb 20, 2024
11 tasks
export function isDefaultQuery(query: SearchQueryVariant): boolean {
return (
isMatchAllQuery(query) ||
(isSimpleQuery(query) && query.query_string.query === '*') ||
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we could add a isDefaultSimplyQuery function to check this * value

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea, updated in 9a6c5c7.

Object.keys(arg.bool).every(
// should be either an empty array or an array with just 1 default query
(d) => {
if (!isPopulatedObject(arg.bool, [d])) return false;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unless I'm misunderstanding this, I don't think this check is needed.
We know d must exist in arg.bool because we've used Object.keys?

If this is the case and this check isn't needed, then this loop could use Object.entries.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right, this is not necessary and more complex than needed; I changed it to just use Object.values because we don't need the key at all. Updated in aa908b2.

import { defaultSimpleQuery } from './simple_query';

describe('isFilterBasedDefaultQuery', () => {
it('should identify filter based default queries', () => {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would separate should identify filter-based default queries and should identify non-default queries

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 updated in 6e6d393.

export function isFilterBasedSimpleQuery(arg: unknown): arg is FilterBasedSimpleQuery {
return (
isPopulatedObject(arg, ['bool']) &&
isPopulatedObject(arg.bool, ['filter']) &&
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just to clarify, is it intentional that this function does not check for other keys in the bool query?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this case yes, we're just interested of a simple query within the filter, not the others options like must and others. I tried to clarify this a bit more in the code, did a bit of renaming and added more comments. The comments include that the purpose of these helpers is to identify stuff in the bool queries the Kibana/EUI search bars expose.

@walterra
Copy link
Contributor Author

walterra commented Mar 7, 2024

@jgowdyelastic @darnautov addressed your comments, ready for another look.

@kibana-ci
Copy link
Collaborator

💚 Build Succeeded

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id before after diff
aiops 431 436 +5
dataVisualizer 623 628 +5
ml 1923 1928 +5
transform 433 438 +5
total +20

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
aiops 402.2KB 402.9KB +676.0B
dataVisualizer 654.0KB 654.7KB +676.0B
transform 402.7KB 391.7KB -11.0KB
total -9.7KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id before after diff
transform 18.4KB 18.4KB -2.0B
Unknown metric groups

API count

id before after diff
@kbn/ml-query-utils 14 28 +14

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @walterra

Copy link
Member

@jgowdyelastic jgowdyelastic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@walterra walterra removed the request for review from qn895 March 8, 2024 08:54
@walterra walterra merged commit 7edaa68 into elastic:main Mar 8, 2024
17 checks passed
@kibanamachine kibanamachine added the backport:skip This commit does not require backporting label Mar 8, 2024
@walterra walterra deleted the ml-176387-fix-functional-tests branch March 8, 2024 09:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport:skip This commit does not require backporting Feature:ML/AIOps ML AIOps features: Change Point Detection, Log Pattern Analysis, Log Rate Analysis :ml release_note:skip Skip the PR/issue when compiling release notes v8.14.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants