Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Index values for Metric Threshold Rule and Inventory Threshold Rule to ALERT_EVALUATION_VALUE field as an array #153877

Closed
Tracked by #145117
simianhacker opened this issue Mar 28, 2023 · 1 comment · Fixed by #154255
Assignees
Labels
enhancement New value added to drive a business result Team: Actionable Observability - DEPRECATED For Observability Alerting and SLOs use "Team:obs-ux-management", for AIops "Team:obs-knowledge"

Comments

@simianhacker
Copy link
Member

We need to modify the Metric Threshold Rule and Inventory Threshold Rules to index the values for each condition in to an array of values for ALERT_EVALUATION_VALUE alert-as-data index. I just verified that this is possible using the diff below with the Metric Threshold Rule

Acceptance Criteria

  • When a rule has multiple conditions, the values should be stored in ALERT_EVALUATION_VALUE as an array with the index corresponding to the condition. The first value in the array should match the first condition.
  • If a condition returns null, the value should be set to null in the array.

Example diff

diff --git a/x-pack/plugins/infra/server/lib/alerting/metric_threshold/metric_threshold_executor.ts b/x-pack/plugins/infra/server/lib/alerting/metric_threshold/metric_threshold_executor.ts
index 28a32a8c461..7fc379f6a4d 100644
--- a/x-pack/plugins/infra/server/lib/alerting/metric_threshold/metric_threshold_executor.ts
+++ b/x-pack/plugins/infra/server/lib/alerting/metric_threshold/metric_threshold_executor.ts
@@ -6,7 +6,7 @@
  */

 import { i18n } from '@kbn/i18n';
-import { ALERT_ACTION_GROUP, ALERT_REASON } from '@kbn/rule-data-utils';
+import { ALERT_ACTION_GROUP, ALERT_EVALUATION_VALUE, ALERT_REASON } from '@kbn/rule-data-utils';
 import { isEqual } from 'lodash';
 import {
   ActionGroupIdsOf,
@@ -79,8 +79,7 @@ type MetricThresholdAlertFactory = (
   reason: string,
   actionGroup: MetricThresholdActionGroup,
   additionalContext?: AdditionalContext | null,
-  threshold?: number | undefined,
-  value?: number | undefined
+  value?: Array<number | null> | undefined
 ) => MetricThresholdAlert;

 export const createMetricThresholdExecutor = (libs: InfraBackendLibs) =>
@@ -117,13 +116,15 @@ export const createMetricThresholdExecutor = (libs: InfraBackendLibs) =>
       id,
       reason,
       actionGroup,
-      additionalContext
+      additionalContext,
+      value
     ) =>
       alertWithLifecycle({
         id,
         fields: {
           [ALERT_REASON]: reason,
           [ALERT_ACTION_GROUP]: actionGroup,
+          [ALERT_EVALUATION_VALUE]: value,
           ...flattenAdditionalContext(additionalContext),
         },
       });
@@ -295,7 +296,14 @@ export const createMetricThresholdExecutor = (libs: InfraBackendLibs) =>
           new Set([...(additionalContext.tags ?? []), ...options.rule.tags])
         );

-        const alert = alertFactory(`${group}`, reason, actionGroupId, additionalContext);
+        const values = alertResults.reduce((acc, result) => {
+          if (result[group]) {
+            acc.push(result[group].currentValue);
+          }
+          return acc;
+        }, [] as Array<number | null>);
+
+        const alert = alertFactory(`${group}`, reason, actionGroupId, additionalContext, values);
         const alertUuid = getAlertUuid(group);
         scheduledActionsCount++;

Sample AAD Document with Multiple Values

{
  "kibana.alert.rule.parameters": {
    "criteria": [
      {
        "aggType": "avg",
        "comparator": ">",
        "threshold": [
          0.01
        ],
        "timeSize": 1,
        "timeUnit": "m",
        "metric": "system.cpu.total.norm.pct"
      },
      {
        "aggType": "rate",
        "comparator": ">",
        "threshold": [
          1
        ],
        "timeSize": 1,
        "timeUnit": "m",
        "metric": "system.network.in.bytes"
      }
    ],
    "sourceId": "default",
    "alertOnNoData": true,
    "alertOnGroupDisappear": true,
    "groupBy": [
      "host.name"
    ]
  },
  "kibana.alert.rule.category": "Metric threshold",
  "kibana.alert.rule.consumer": "alerts",
  "kibana.alert.rule.execution.uuid": "1231f14e-734c-4245-8b36-b37423c6e2e5",
  "kibana.alert.rule.name": "Test MTR",
  "kibana.alert.rule.producer": "infrastructure",
  "kibana.alert.rule.rule_type_id": "metrics.alert.threshold",
  "kibana.alert.rule.uuid": "b191ded0-cd78-11ed-992d-05bd7cd123f9",
  "kibana.space_ids": [
    "default"
  ],
  "kibana.alert.rule.tags": [],
  "@timestamp": "2023-03-28T14:56:19.405Z",
  "kibana.alert.reason": """system.cpu.total.norm.pct is 41.7% in the last 1 min for host-0. Alert when > 1%.
system.network.in.bytes is 1,000 in the last 1 min for host-0. Alert when > 1.""",
  "kibana.alert.action_group": "metrics.threshold.fired",
  "kibana.alert.evaluation.value": [
    0.4166666666666667,
    1000
  ],
  "host.name": "host-0",
  "host.mac": [
    "00-00-5E-00-53-23",
    "00-00-5E-00-53-24"
  ],
  "labels.eventId": "event-0",
  "labels.groupId": "group-0",
  "tags": [
    "group-0",
    "event-0"
  ],
  "kibana.alert.duration.us": 0,
  "kibana.alert.time_range": {
    "gte": "2023-03-28T14:56:19.405Z"
  },
  "kibana.alert.instance.id": "host-0",
  "kibana.alert.start": "2023-03-28T14:56:19.405Z",
  "kibana.alert.uuid": "16752793-9be6-4c1f-9b7f-ce87bc408f2f",
  "kibana.alert.status": "active",
  "kibana.alert.workflow_status": "open",
  "event.kind": "signal",
  "event.action": "open",
  "kibana.version": "8.8.0",
  "kibana.alert.flapping": false
}
@simianhacker simianhacker added enhancement New value added to drive a business result Team: Actionable Observability - DEPRECATED For Observability Alerting and SLOs use "Team:obs-ux-management", for AIops "Team:obs-knowledge" labels Mar 28, 2023
@elasticmachine
Copy link
Contributor

Pinging @elastic/actionable-observability (Team: Actionable Observability)

@maryam-saeidi maryam-saeidi self-assigned this Mar 30, 2023
maryam-saeidi added a commit that referenced this issue Apr 20, 2023
…154255)

Closes #153877

## Summary

This PR adds a new field called `kibana.alert.evaluation.values` to the
alert document for metric threshold and inventory rules. This is an
array of numbers but depending on the result of the rule execution, the
value might be `null` too.


![image](https://user-images.githubusercontent.com/12370520/230380396-fcfa10d8-a119-497b-bd94-9f567ecb8fc5.png)

We want to use this result in the metric threshold alert details page,
so I checked whether this value can be retrieved correctly there or not:

![image](https://user-images.githubusercontent.com/12370520/230380867-3a0520fd-687c-4d88-8161-278abfe8fc88.png)

**Note**
I will add tests later, I would like to get feedback about the
implementation first.

## 🧪 How to test
- Add xpack.observability.unsafe.alertDetails.metrics.enabled: true to
the Kibana config
- Create a metric threshold and inventory rule that generates an alert
- Check the alert document for the `kibana.alert.evaluation.values`
field, it should be an array with the result of evaluation for the
related criteria
- If you are using metricbeat, stop it so the value of evaluation will
be null
- Go to the alert details page, you should be able to see the main chart
even when the evaluation value is null
- Check the alert document for the `kibana.alert.evaluation.values`
field, it should be an array including a null value
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New value added to drive a business result Team: Actionable Observability - DEPRECATED For Observability Alerting and SLOs use "Team:obs-ux-management", for AIops "Team:obs-knowledge"
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants