Fix issue with incorrect policy response attachment.

[pjhampton]

Summary

This is a tricky PR to explain - the (sometimes) incorrect policy response has been being attached for 7.14.0 + 7.15.0 Endpoints.

Background

An EP telemetry document is built of 4 parts:

• Top-level information. eg EP ID, EP Version, Package Version, etc
• Metrics. eg CPU, Mem etc
• Policy Configuration
• Policy Response (if not 100% successful and Policy config exists)

To get all this information we have to.

1. Read EP metrics for last 24h
--> 2. Find all active Fleet agents in the last 24 hours
----> 3. Load *Agent policies*
------> 4. Find and load *EP Package Policies* from agent policies
--------> 5. Join Policy Configs with Endpoints via Elastic Agent (Step 1)
----------> 6. Load Policy Responses in last 24 hours
------------> 7. Join Policy Responses to Endpoint (if policy config exists)

The reason for this complexity is that there is no policy config reference in the EP Metrics document. To get this to work I look up all the agent policies and look up the EP package policies. The original assumption aggregated the policy document from the .ds-metrics-endpoint.policy* datastream on the policy id "terms": { "field": "Endpoint.policy.applied.id" }

GET /.ds-metrics-endpoint.policy*/_search?expand_wildcards=open,hidden
{
  "size": 0,
  "query": {
    "match": {
      "Endpoint.policy.applied.status": "failure"
    }
  },
  "aggs": {
    "policy_responses": {
      "terms": {
        "field": "Endpoint.policy.applied.id",
        "size": 10
      },
      "aggs": {
        "latest_response": {
          "top_hits": {
            "size": 1,
            "sort": [
              {
                "@timestamp": {
                  "order": "desc"
                }
              }
            ]
          }
        }
      }
    }
  }
}

However, this was a poor assumption. The reason for this is the policy config is applied to a fleet of endpoints, rather than a specific endpoint. The above has been changed to aggregate on the Endpoint ID "terms": { "field": "agent.id" } which conveniently is present on the policy response document.

GET /.ds-metrics-endpoint.policy*/_search?expand_wildcards=open,hidden
{
  "size": 0,
  "query": {
    "match": {
      "Endpoint.policy.applied.status": "failure"
    }
  },
  "aggs": {
    "policy_responses": {
      "terms": {
        "field": "agent.id",
        "size": 10
      },
      "aggs": {
        "latest_response": {
          "top_hits": {
            "size": 1,
            "sort": [
              {
                "@timestamp": {
                  "order": "desc"
                }
              }
            ]
          }
        }
      }
    }
  }
}

Checklist

• Documentation was added for features that require explanation or tutorials

For maintainers

• This was checked for breaking API changes and was labeled appropriately

[pjhampton]

@elasticmachine merge upstream

[pjhampton]

@elasticmachine merge upstream

[pjhampton]

@elasticmachine merge upstream

[pjhampton]

@elasticmachine merge upstream

[donaherc]

This is great thanks. The context writeup on this issue is superlative as well, great for context about how this particularly complicated telemetry document is assembled.

[pjhampton]

@elasticmachine merge upstream

[kibanamachine]

💛 Build succeeded, but was flaky

• continuous-integration/kibana-ci/pull-request

• Commit: 91b2e5d

• Storybooks Preview

• Flaky suites:

  • xpack-securitySolutionCypressChrome

---

Test Failures

Kibana Pipeline / general / Allows the rule to be duplicated from the edit screen.indicator match Detection rules, Indicator Match Duplicates the indicator rule Allows the rule to be duplicated from the edit screen

Link to Jenkins

Stack Trace

Failed Tests Reporter:
  - Test has not failed recently on tracked branches

AssertionError: Timed out retrying after 60000ms: expected '<button.euiContextMenuItem>' to be 'visible'

This element `<button.euiContextMenuItem>` is not visible because its parent `<div>` has CSS property: `visibility: hidden`
    at Context.eval (http://localhost:6121/__cypress/tests?p=cypress/integration/detection_rules/indicator_match_rule.spec.ts:31238:58)

---

Metrics [docs]

✅ unchanged

History

• 💔 Build #157626 failed 1b27870
• 💛 Build #157489 was flaky 20c7ebe
• 💚 Build #156655 succeeded 51cc184
• 💛 Build #156568 was flaky eb447cc
• 💔 Build #156460 failed 82a8628

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @pjhampton

[kibanamachine]

💚 Backport successful

Status	Branch	Result
✅	7.x
✅	7.15

The backport PRs will be merged automatically after passing CI.