add collection data stream and ilm policy #118
Conversation
| @@ -0,0 +1,15 @@ | |||
| { | |||
There was a problem hiding this comment.
What's the difference between this ilm_policy and this one https://github.com/elastic/endpoint-package/pull/118/files#diff-4dd7edc183ee15694b9a686b280db66ea3864789768bb95daa6c4d56140a49d5R1
There was a problem hiding this comment.
Ah this should be deleted
| } | ||
| } | ||
| } | ||
| } |
There was a problem hiding this comment.
I think we need a delete phase, otherwise the indices will be rolled over but not actually deleted. In my testing, I did something like this:
PUT /_ilm/policy/endpoint.diagnostic.collection
{
"policy": {
"phases": {
"hot": {
"min_age": "0ms",
"actions": {
"rollover": {
"max_size": "1gb",
"max_age": "7d",
"max_docs": 10000
}
}
},
"delete": {
"min_age": "10m",
"actions": {
"delete": {
}
}
}
}
}
}
That should mean that the rolled-over index is deleted 10m after it's rotated. Comparing to your version, I also have a limitation of max_docs. Not sure we need it, but maybe good for testing.
There was a problem hiding this comment.
thanks @tsg I updated the policy
…-new-collection-data-stream
| @@ -0,0 +1,28 @@ | |||
| --- | |||
There was a problem hiding this comment.
Just want to double check that this is the intent but this mapping is pretty reduced. The only fields that will be searchable within ES are @timestamp, data_stream, ecs.version, and the event.* fields. Is that what we want?
There was a problem hiding this comment.
@jonathan-buttner this is the desired functionality.
This will be read by a telemetry server and it will search by event.ingested. We don't need to index any other fields as we'll only need to search for the latest docs.
The Kibana side changes are ready, the PRs are up. I've tested everything together locally. My plan is to merge the Fleet changes as soon as I get approval and then I'll merge this |
Co-authored-by: Kevin Logan <klogan@endgame.com>
This PR adds the new
data streamthat will be used for diagnostic alerts.It contains:
hiddenfield which will make the diagnosticdata streamhiddenilm_policywhich will be aggressiveThese fields are now supported in the package-spec:
elastic/package-spec#91
elastic/package-spec#90
And released via the package-registry:
https://github.com/elastic/package-registry/releases/tag/v0.15.0
It's been tested locally via new changes in Fleet package installation.
elastic/kibana#85492
elastic/kibana#85703