[ML] APM modules configs for RUM Javascript and NodeJS #53792

darnautov · 2019-12-24T12:54:36Z

Summary

Add APM modules configurations for RUM JS and NodeJS
Custom URLs support for APM

Checklist

This was checked for cross-browser compatibility, including a check against IE11
Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
~~Documentation was added for features that require explanation or tutorials~~
Unit or functional tests were updated or added to match the most common scenarios
~~This was checked for keyboard-only and screenreader accessibility~~

elasticmachine · 2019-12-24T12:54:38Z

Pinging @elastic/ml-ui (:ml)

kibanamachine · 2019-12-24T14:12:17Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request
Commit: 5ad8bab356055b199f701fa863e76468531747cd

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

blaklaybul

requesting a few minor changes

blaklaybul · 2019-12-24T15:30:24Z

x-pack/legacy/plugins/ml/server/models/data_recognizer/modules/apm_jsbase/manifest.json

+{
+  "id": "apm_jsbase",
+  "title": "APM: RUM Javascript",
+  "description": "Detect anomalies in spans and errors",


Detect problematic spans and identify user agents that are potentially causing issues.

Fixed in 144c0ba

blaklaybul · 2019-12-24T15:36:05Z

x-pack/legacy/plugins/ml/server/models/data_recognizer/modules/apm_nodejs/manifest.json

+{
+  "id": "apm_nodejs",
+  "title": "APM: NodeJS",
+  "description": "Detect anomalies in transactions, spans, and errors",


Detect abnormal traces, anomalous spans, and identify periods of decreased throughput.

Fixed in 144c0ba

.../ml/server/models/data_recognizer/modules/apm_nodejs/ml/abnormal_trace_durations_nodejs.json

kibanamachine · 2019-12-24T16:53:14Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: c6039ecddfa888a594473d11b848b1533ebf3216

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

darnautov · 2020-01-01T17:22:43Z

@elasticmachine merge upstream

walterra

Code LGTM

alvarezmelissa87

LGTM ⚡️

...er/models/data_recognizer/modules/apm_jsbase/ml/datafeed_abnormal_span_durations_jsbase.json

...a_recognizer/modules/apm_jsbase/ml/datafeed_anomalous_error_rate_for_user_agents_jsbase.json

...r/models/data_recognizer/modules/apm_jsbase/ml/datafeed_high_count_by_user_agent_jsbase.json

x-pack/legacy/plugins/ml/server/models/data_recognizer/modules/apm_nodejs/manifest.json

...er/models/data_recognizer/modules/apm_nodejs/ml/datafeed_abnormal_span_durations_nodejs.json

...erver/models/data_recognizer/modules/apm_nodejs/ml/datafeed_decreased_throughput_nodejs.json

...erver/models/data_recognizer/modules/apm_jsbase/ml/datafeed_decreased_throughput_jsbase.json

...erver/models/data_recognizer/modules/apm_nodejs/ml/datafeed_decreased_throughput_nodejs.json

x-pack/legacy/plugins/ml/server/models/data_recognizer/modules/apm_jsbase/manifest.json

...erver/models/data_recognizer/modules/apm_jsbase/ml/datafeed_decreased_throughput_jsbase.json

...gins/ml/server/models/data_recognizer/modules/apm_jsbase/ml/decreased_throughput_jsbase.json

...s/ml/server/models/data_recognizer/modules/apm_jsbase/ml/abnormal_span_durations_jsbase.json

...gins/ml/server/models/data_recognizer/modules/apm_jsbase/ml/decreased_throughput_jsbase.json

blaklaybul

Looks good so far! Some comments:

remove scroll_size and chunking_config parameters.
update detector descriptions

blaklaybul · 2020-01-06T15:17:03Z

...s/ml/server/models/data_recognizer/modules/apm_jsbase/ml/abnormal_span_durations_jsbase.json

+    "bucket_span": "15m",
+    "detectors": [
+      {
+        "detector_description": "high_mean(\"span.duration.us\") partitionfield=\"span.type\"",


"description": "increased span duration"

blaklaybul · 2020-01-06T15:17:30Z

...odels/data_recognizer/modules/apm_jsbase/ml/anomalous_error_rate_for_user_agents_jsbase.json

+    "bucket_span": "15m",
+    "detectors": [
+      {
+        "detector_description": "high_non_zero_count partitionfield=\"user_agent.name\"",


"description": "high error rate for user agent"

blaklaybul · 2020-01-06T15:17:59Z

...gins/ml/server/models/data_recognizer/modules/apm_jsbase/ml/decreased_throughput_jsbase.json

+    "bucket_span": "15m",
+    "detectors": [
+      {
+        "detector_description": "low_count",


"description": "low throughput"

blaklaybul · 2020-01-06T15:18:23Z

.../ml/server/models/data_recognizer/modules/apm_jsbase/ml/high_count_by_user_agent_jsbase.json

+    "bucket_span": "15m",
+    "detectors": [
+      {
+        "detector_description": "high_non_zero_count partitionfield=\"user_agent.name\"",


"description": "high request rate for user agent"

blaklaybul · 2020-01-06T15:18:55Z

...s/ml/server/models/data_recognizer/modules/apm_nodejs/ml/abnormal_span_durations_nodejs.json

+    "bucket_span": "15m",
+    "detectors": [
+      {
+        "detector_description": "high_mean(\"span.duration.us\") partitionfield=\"span.type\"",


"description": "increased span duration"

blaklaybul · 2020-01-06T15:19:15Z

.../ml/server/models/data_recognizer/modules/apm_nodejs/ml/abnormal_trace_durations_nodejs.json

+    "bucket_span": "15m",
+    "detectors": [
+      {
+        "detector_description": "high_mean(\"transaction.duration.us\") by \"transaction.name\"",


"description": "increased trace duration"

blaklaybul · 2020-01-06T15:19:42Z

...gins/ml/server/models/data_recognizer/modules/apm_nodejs/ml/decreased_throughput_nodejs.json

+    "bucket_span": "15m",
+    "detectors": [
+      {
+        "detector_description": "low_count",


"description": "low throughput"

blaklaybul

Detector descriptions look good, but you should be able to remove the chunking_config parameter from the 2 decreased_throughput datafeeds as well

darnautov · 2020-01-07T09:39:24Z

@blaklaybul 374c4eb ?

blaklaybul

LGTM

blaklaybul

additional changes to custom URLs and datafeed filter.

blaklaybul · 2020-01-07T16:01:26Z

...r/models/data_recognizer/modules/apm_jsbase/ml/datafeed_high_count_by_user_agent_jsbase.json

+    "bool": {
+      "must": [
+        { "bool": { "filter": { "term": { "agent.name": "js-base" } } } },
+        { "bool": { "filter": [{ "exists": { "field": "user_agent.name" } }] } }


please add an additional filter for transaction to the must:

{"bool": {"filter": {"term": {"processor.event": "transaction"}}}}

...gins/ml/server/models/data_recognizer/modules/apm_nodejs/ml/decreased_throughput_nodejs.json

...gins/ml/server/models/data_recognizer/modules/apm_jsbase/ml/decreased_throughput_jsbase.json

darnautov · 2020-01-08T13:11:31Z

@elasticmachine merge upstream

peteharverson

Tested latest edits and LGTM

blaklaybul

Latests changes lgtm

darnautov · 2020-01-10T17:27:42Z

@elasticmachine merge upstream

darnautov · 2020-01-10T20:22:52Z

@elasticmachine merge upstream

kibanamachine · 2020-01-10T21:48:04Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: fe6058a

History

💔 Build #19354 failed ba69c89
💚 Build #18687 succeeded 7f465fe
💚 Build #18671 succeeded 2d1fe5c
💚 Build #18317 succeeded 374c4eb
💚 Build #18115 succeeded ebe6353

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

* [ML] apm modules * [ML] apm modules * [ML] update mocha test * [ML] fix config * [ML] single line JSON formatting for queries * [ML] remove an empty path component with a trailing slash * [ML] change detector descriptions, remove scroll size * [ML] remove chunking_config from datafeeds * [ML] fix configs Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

* master: (69 commits) [Graph] Fix various a11y issues (elastic#54097) Add ApplicationService app status management (elastic#50223) logs in one time (elastic#54447) Deprecate using `elasticsearch.ssl.certificate` without `elasticsearch.ssl.key` and vice versa (elastic#54392) [Optimizer] Fix a stack overflow with watch_cache when it attempts to delete very large folders. (elastic#54457) Security - Role Mappings UI (elastic#53620) [SIEM] [Detection engine] Permission II (elastic#54292) Allow User to Cleanup Repository from UI (elastic#53047) [Detection engine] Some UX for rule creation (elastic#54471) share specific instances of some ui packages (elastic#54079) [ML] APM modules configs for RUM Javascript and NodeJS (elastic#53792) [APM] Delay rendering invalid license notification (elastic#53924) [Graph] Improve error message on graph requests (elastic#54230) [ILM] Kibana should allow a min_age setting of 0ms in ILM policy phases (elastic#53719) Unit Tests for common/lib (elastic#53736) [Graph] Only show explorable fields (elastic#54101) remove linting rule exception for markdown (elastic#54232) [Monitoring] Fetch shard data more efficiently (elastic#54028) [Maps] Add hiddenLayers option to embeddable map input (elastic#54355) Pass termOrder and hasTermsAgg properties to serializeThresholdWatch function (elastic#54391) ...

* [ML] apm modules * [ML] apm modules * [ML] update mocha test * [ML] fix config * [ML] single line JSON formatting for queries * [ML] remove an empty path component with a trailing slash * [ML] change detector descriptions, remove scroll size * [ML] remove chunking_config from datafeeds * [ML] fix configs Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

darnautov added release_note:enhancement :ml Feature:Anomaly Detection ML anomaly detection v8.0.0 v7.6.0 labels Dec 24, 2019

darnautov requested review from blaklaybul, alvarezmelissa87 and peteharverson December 24, 2019 12:54

darnautov requested a review from a team as a code owner December 24, 2019 12:54

darnautov self-assigned this Dec 24, 2019

blaklaybul suggested changes Dec 24, 2019

View reviewed changes

darnautov requested a review from walterra December 27, 2019 12:52

darnautov force-pushed the ML-new-apm-jobs branch from c6039ec to 144c0ba Compare December 30, 2019 12:10

darnautov requested a review from blaklaybul December 30, 2019 12:11

walterra approved these changes Jan 2, 2020

View reviewed changes

alvarezmelissa87 approved these changes Jan 2, 2020

View reviewed changes

peteharverson requested changes Jan 2, 2020

View reviewed changes

peteharverson reviewed Jan 2, 2020

View reviewed changes

...gins/ml/server/models/data_recognizer/modules/apm_jsbase/ml/decreased_throughput_jsbase.json Show resolved Hide resolved

peteharverson reviewed Jan 2, 2020

View reviewed changes

...s/ml/server/models/data_recognizer/modules/apm_jsbase/ml/abnormal_span_durations_jsbase.json Outdated Show resolved Hide resolved

peteharverson reviewed Jan 2, 2020

View reviewed changes

...gins/ml/server/models/data_recognizer/modules/apm_jsbase/ml/decreased_throughput_jsbase.json Outdated Show resolved Hide resolved

darnautov added 5 commits January 6, 2020 10:22

[ML] apm modules

a5f509d

[ML] apm modules

c8505f5

[ML] update mocha test

c8b1ef7

[ML] fix config

9a015c3

[ML] single line JSON formatting for queries

f40d0a2

darnautov force-pushed the ML-new-apm-jobs branch from cbf8ddf to f40d0a2 Compare January 6, 2020 09:50

[ML] remove an empty path component with a trailing slash

9f8347d

darnautov requested a review from peteharverson January 6, 2020 12:18

blaklaybul suggested changes Jan 6, 2020

View reviewed changes

[ML] change detector descriptions, remove scroll size

ebe6353

darnautov requested a review from blaklaybul January 6, 2020 16:21

blaklaybul suggested changes Jan 6, 2020

View reviewed changes

[ML] remove chunking_config from datafeeds

374c4eb

darnautov requested a review from blaklaybul January 7, 2020 09:39

blaklaybul approved these changes Jan 7, 2020

View reviewed changes

blaklaybul suggested changes Jan 7, 2020

View reviewed changes

[ML] fix configs

2d1fe5c

darnautov requested a review from blaklaybul January 8, 2020 12:06

Merge branch 'master' into ML-new-apm-jobs

7f465fe

peteharverson approved these changes Jan 9, 2020

View reviewed changes

blaklaybul approved these changes Jan 10, 2020

View reviewed changes

Merge branch 'master' into ML-new-apm-jobs

ba69c89

Merge branch 'master' into ML-new-apm-jobs

fe6058a

darnautov merged commit 51c1a8f into elastic:master Jan 10, 2020

darnautov mentioned this pull request Jan 13, 2020

[7.x] [ML] APM modules configs for RUM Javascript and NodeJS (#53792) #54553

Merged

[ML] APM modules configs for RUM Javascript and NodeJS #53792

[ML] APM modules configs for RUM Javascript and NodeJS #53792

Conversation

darnautov commented Dec 24, 2019 • edited Loading

Summary

Checklist

elasticmachine commented Dec 24, 2019

kibanamachine commented Dec 24, 2019

💔 Build Failed

blaklaybul left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kibanamachine commented Dec 24, 2019

💚 Build Succeeded

darnautov commented Jan 1, 2020

walterra left a comment

Choose a reason for hiding this comment

alvarezmelissa87 left a comment

Choose a reason for hiding this comment

blaklaybul left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

blaklaybul left a comment

Choose a reason for hiding this comment

darnautov commented Jan 7, 2020

blaklaybul left a comment

Choose a reason for hiding this comment

blaklaybul left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

darnautov commented Jan 8, 2020

peteharverson left a comment

Choose a reason for hiding this comment

blaklaybul left a comment

Choose a reason for hiding this comment

darnautov commented Jan 10, 2020

darnautov commented Jan 10, 2020

kibanamachine commented Jan 10, 2020

💚 Build Succeeded

History

darnautov commented Dec 24, 2019 •

edited

Loading

blaklaybul left a comment •

edited

Loading