added telemetry with most common error from agent logs

[juliaElastic]

Summary

Closes https://github.com/elastic/ingest-dev/issues/1261

Merged: elasticsearch change to give kibana_system the missing privilege to read logs-elastic_agent* indices.

Top 3 most common errors in the Elastic Agent logs

Added most common elastic-agent and fleet-server logs to telemetry.

Using a query of message field using sampler and categorize text aggregation. This is a workaround as we can't directly do aggregation on message field.

GET logs-elastic_agent*/_search
{
    "size": 0,
    "query": {
        "bool": {
            "must": [
                {
                    "term": {
                        "log.level": "error"
                    }
                },
                {
                    "range": {
                        "@timestamp": {
                            "gte": "now-1h"
                        }
                    }
                }
            ]
        }
    },
    "aggregations": {
        "message_sample": {
            "sampler": {
                "shard_size": 200
            },
            "aggs": {
                "categories": {
                    "categorize_text": {
                        "field": "message",
                        "size": 10
                    }
                }
            }
        }
    }
}

Tested with latest Elasticsearch snapshot, and verified that the logs are added to telemetry:

   {
      "agent_logs_top_errors": [
         "failed to dispatch actions error failed reloading q q q nil nil config failed reloading artifact config for composed snapshot.downloader failed to generate snapshot config failed to detect remote snapshot repo proceeding with configured not an agent uri",
         "fleet-server stderr level info time message No applicable limit for agents using default \\n level info time message No applicable limit for agents using default \\n",
         "stderr panic close of closed channel n ngoroutine running Stop"
      ],
      "fleet_server_logs_top_errors": [
         "Dispatch abort response",
         "error while closing",
         "failed to take ownership"
      ]
   }

Did some measurements locally, and the query took a few ms only. I'll try to check with larger datasets in elastic agent logs too.

Checklist

• Unit or functional tests were updated or added to match the most common scenarios

[juliaElastic]

@elasticmachine merge upstream

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[juliaElastic]

@elasticmachine merge upstream

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: 801302c

Metrics [docs]

Unknown metric groups

ESLint disabled in files

id	before	after	diff
osquery	1	2	+1

ESLint disabled line counts

id	before	after	diff
enterpriseSearch	19	21	+2
fleet	59	65	+6
osquery	109	115	+6
securitySolution	442	448	+6
total			+20

Total ESLint disabled count

id	before	after	diff
enterpriseSearch	20	22	+2
fleet	68	74	+6
osquery	110	117	+7
securitySolution	519	525	+6
total			+21

History

• 💔 Build #90916 failed 8bc53e4
• 💛 Build #90303 was flaky 42a0d49
• 💚 Build #90209 succeeded 5cb4a12

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @juliaElastic

[kpollich]

LGTM! Awesome change.

[jakommo]

For reference, this actually made it into 8.6.0.