[Fleet] add ilm policy per data stream #85492

nnamdifrankie · 2020-12-09T23:42:48Z

Summary

This PR adds the installation flow for ilm_policy per data_stream. This feature will be used to define a custom ilm_policy for a data_stream that will override the one taken based on type.

Example package PR that will use this field: elastic/endpoint-package#118

A screenshot showing the ILM policy that's installed from the Endpoint package.

Here's the Index Template showing that it's lifecycle points to the installed ILM policy

add ilm_policy for data streams and override the default type

Checklist

Delete any items that are not applicable to this PR.

Unit or functional tests were updated or added to match the most common scenarios

kevinlog · 2020-12-10T00:15:23Z

x-pack/plugins/fleet/server/services/epm/packages/_install_package.ts

@@ -30,6 +30,7 @@ import { deleteKibanaSavedObjectsAssets } from './remove';
 import { installTransform } from '../elasticsearch/transform/install';
 import { createInstallation, saveKibanaAssetsRefs, updateVersion } from './install';
 import { saveArchiveEntries } from '../archive/save_to_es';
+import { installIlmForDataStream } from '../elasticsearch/datastream_ilm/install';


is this file missing from the commit?

yes, updated

kevinlog · 2020-12-10T00:15:33Z

x-pack/plugins/fleet/server/services/epm/packages/remove.ts

@@ -24,6 +24,7 @@ import { packagePolicyService, appContextService } from '../..';
 import { splitPkgKey } from '../registry';
 import { deletePackageCache } from '../archive';
 import { removeArchiveEntries } from '../archive/save_to_es';
+import { deleteIlms } from '../elasticsearch/datastream_ilm/remove';


same here, is this file missing from the commit?

yes, updated

ruflin · 2020-12-10T12:18:32Z

Again, would have preferred to have 2 separate PR's for ilm policy and hidden to keep the features separate.

ruflin · 2020-12-10T12:22:34Z

What is the name of the ILM policies that are loaded?

…tion-ilm-policy-and-hidden

kevinlog · 2020-12-11T15:34:15Z

@ruflin

What is the name of the ILM policies that are loaded?

Currently, the one included in the data_stream for the endpoint package will be this: logs-endpoint.collection-diagnostics

Screenshot from running the code + package locally:

There's another example in a couple of the tests.
https://github.com/elastic/kibana/pull/85492/files#diff-fa07cac51b6a49bf1e4824bc2250c9a77dac6c7d6b0a56020f559ef1ff9be25fR463

I don't have a strong opinion on the naming, happy to change it.

cc/ @tsg

elasticmachine · 2020-12-11T19:11:56Z

Pinging @elastic/ingest-management (Team:Ingest Management)

kevinlog · 2020-12-11T19:13:19Z

@ruflin

Again, would have preferred to have 2 separate PR's for ilm policy and hidden to keep the features separate.

I opened this PR to separate ilm_policy and hidden #85703

ruflin · 2020-12-14T08:20:43Z

@kevinlog On the naming, I would assume the ilm policy has a similar / same name as the data stream(s) it applies to?

kevinlog · 2020-12-14T12:37:47Z

@elasticmachine merge upstream

skh

Happy with the code as long as @ruflin is happy with the implemented functionality.

kevinlog · 2020-12-14T13:23:50Z

@ruflin

@ kevinlog On the naming, I would assume the ilm policy has a similar / same name as the data stream(s) it applies to?

The names are similar, however we can change it so that they are the same.

Index template: logs-endpoint.diagnostics.collection

current ILM Policy name: logs-endpoint.collection-diagnostics
suggested ILM Policy name: logs-endpoint.diagnostics-collection

Let me know if the above looks good. Happy to make adjustments

ruflin · 2020-12-14T13:59:51Z

I think from a user perspective it would be easier if the name would be indentical so logs-endpoint.diagnostics.collection but I guess I miss why there is a variation.

kevinlog · 2020-12-14T14:04:47Z

@ruflin

I think from a user perspective it would be easier if the name would be indentical so logs-endpoint.diagnostics.collection but I guess I miss why there is a variation.

I tend to agree with you, however the existing ilm_policy's I see all use - as opposed to . I don't know if there's a reason.

ruflin · 2020-12-14T14:30:15Z

@kevinlog Are these ILM policies related to Packages / New Indexing Strategy?

kevinlog · 2020-12-14T14:33:52Z

@ruflin

@ kevinlog Are these ILM policies related to Packages / New Indexing Strategy?

yes, these ILM policies will be shipped with the package, here's an example with the Endpoint package: elastic/endpoint-package#118

kevinlog · 2020-12-14T16:03:59Z

update from the Fleet sync - we'll merge this as-is. We can continue to talk about the naming after FF and before 7.11 ships and make adjustments.

cc/ @ph @ruflin @skh

jonathan-buttner

Some questions about the regex patterns.

jonathan-buttner · 2020-12-14T18:08:30Z

x-pack/plugins/fleet/server/services/epm/elasticsearch/datastream_ilm/install.ts

+}
+
+const isDataStreamIlm = (path: string) => {
+  return new RegExp('(?<package>.*)/data_stream/(?<dataset>.*)/elasticsearch/ilm/*.*').test(path);


nit: Do we need the ?<package> or the ?<dataset> ? I believe those are so we can extract out those specific fields.

Do we need to escape all the /?
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Regular_Expressions

Similarly, if you're writing a regular expression literal and need to match a slash ("/"), you need to escape that (otherwise, it terminates the pattern). For instance, to search for the string "/example/" followed by one or more alphabetic characters, you'd use //example/[a-z]+/i—the backslashes before each slash make them literal.

I was playing around with it here:
https://regex101.com/r/GLnjJN/1
https://regex101.com/r/GLnjJN/2

I think we might need one of these:

(?<package>.*)\/data_stream\/(?<dataset>.*)\/elasticsearch\/ilm\/.*

.*\/data_stream\/.*\/elasticsearch\/ilm\/.*

Do we need? No, probably not. (?<package>.*) vs .* is adding a named capture group.

It doesn't look like we access or use the groups, so that's unnecessary. Adding the label, though, somewhat acts as in-regex-documentation of what we're looking at. I do like that.

We do not need to escape the slashes here because it is a new Regexp constructor using a string. Not using slash-literals.

jonathan-buttner · 2020-12-14T18:15:20Z

x-pack/plugins/fleet/server/services/epm/elasticsearch/datastream_ilm/install.ts

+};
+
+const isDatasetIlm = (path: string, datasetName: string) => {
+  return new RegExp(`(?<package>.*)/data_stream\\/${datasetName}/elasticsearch/ilm/*.*`).test(path);


Same questions about groups and escaping as above

…tion-ilm-policy-and-hidden

kibanamachine · 2020-12-14T21:15:09Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: d058bb4

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`fleet`	950.4KB	950.5KB	+48.0B

Distributable file count

id	before	after	diff
`default`	47130	47892	+762

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`fleet`	376.5KB	376.5KB	+71.0B

History

💔 Build #94136 failed dfa2410
💚 Build #93971 succeeded 109bb7f
💚 Build #93798 succeeded 6f74373
💚 Build #93781 succeeded f6433aa
💔 Build #93777 failed 204d903

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

Co-authored-by: kevinlog <kevin.logan@elastic.co> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

Co-authored-by: kevinlog <kevin.logan@elastic.co> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: nnamdifrankie <56440728+nnamdifrankie@users.noreply.github.com> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

add hidden datastream and ilm policy

6e31537

kevinlog reviewed Dec 10, 2020

View reviewed changes

add files

128db0f

ruflin added the Team:Fleet Team label for Observability Data Collection Fleet team label Dec 10, 2020

kevinlog added 4 commits December 11, 2020 09:15

fix tests, some types

21b498e

Merge branch 'master' of github.com:kevinlog/kibana into add-installa…

204d903

…tion-ilm-policy-and-hidden

fix type

e1b3f5c

unskip tests

f6433aa

remove hidden field feature

0b528da

kevinlog changed the title ~~add hidden datastream and ilm policy~~ [Fleet] add ilm policy per data stream Dec 11, 2020

kevinlog marked this pull request as ready for review December 11, 2020 19:11

kevinlog requested a review from a team December 11, 2020 19:11

kevinlog added release_note:skip Skip the PR/issue when compiling release notes v7.11.0 labels Dec 11, 2020

add second ilm_policy per data stream

6f74373

kevinlog mentioned this pull request Dec 11, 2020

add collection data stream and ilm policy elastic/endpoint-package#118

Merged

ruflin requested a review from skh December 14, 2020 08:23

Merge branch 'master' into add-installation-ilm-policy-and-hidden

109bb7f

skh approved these changes Dec 14, 2020

View reviewed changes

skh requested a review from ruflin December 14, 2020 13:14

kevinlog requested a review from jonathan-buttner December 14, 2020 14:44

kevinlog requested a review from pzl December 14, 2020 16:22

jonathan-buttner reviewed Dec 14, 2020

View reviewed changes

kevinlog added 2 commits December 14, 2020 13:34

Merge branch 'master' of github.com:kevinlog/kibana into add-installa…

dfa2410

…tion-ilm-policy-and-hidden

Merge branch 'master' of github.com:kevinlog/kibana into add-installa…

d058bb4

…tion-ilm-policy-and-hidden

jonathan-buttner approved these changes Dec 14, 2020

View reviewed changes

kevinlog merged commit 23c5daa into elastic:master Dec 14, 2020

kevinlog added a commit to kevinlog/kibana that referenced this pull request Dec 14, 2020

[Fleet] add ilm policy per data stream (elastic#85492)

5189eac

Co-authored-by: kevinlog <kevin.logan@elastic.co> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

kevinlog mentioned this pull request Dec 14, 2020

[7.x] [Fleet] add ilm policy per data stream (#85492) #85865

Merged

kevinlog mentioned this pull request Jan 18, 2021

Fleet EPM Supported Assets for integration #88499

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Fleet] add ilm policy per data stream #85492

[Fleet] add ilm policy per data stream #85492

nnamdifrankie commented Dec 9, 2020 •

edited by kevinlog

Loading

kevinlog Dec 10, 2020

nnamdifrankie Dec 10, 2020

kevinlog Dec 10, 2020

nnamdifrankie Dec 10, 2020

ruflin commented Dec 10, 2020

ruflin commented Dec 10, 2020

kevinlog commented Dec 11, 2020 •

edited

Loading

elasticmachine commented Dec 11, 2020

kevinlog commented Dec 11, 2020

ruflin commented Dec 14, 2020

kevinlog commented Dec 14, 2020

skh left a comment

kevinlog commented Dec 14, 2020

ruflin commented Dec 14, 2020

kevinlog commented Dec 14, 2020

ruflin commented Dec 14, 2020

kevinlog commented Dec 14, 2020

kevinlog commented Dec 14, 2020

jonathan-buttner left a comment

jonathan-buttner Dec 14, 2020

pzl Dec 14, 2020

jonathan-buttner Dec 14, 2020

kibanamachine commented Dec 14, 2020

[Fleet] add ilm policy per data stream #85492

[Fleet] add ilm policy per data stream #85492

Conversation

nnamdifrankie commented Dec 9, 2020 • edited by kevinlog Loading

Summary

Checklist

kevinlog Dec 10, 2020

Choose a reason for hiding this comment

nnamdifrankie Dec 10, 2020

Choose a reason for hiding this comment

kevinlog Dec 10, 2020

Choose a reason for hiding this comment

nnamdifrankie Dec 10, 2020

Choose a reason for hiding this comment

ruflin commented Dec 10, 2020

ruflin commented Dec 10, 2020

kevinlog commented Dec 11, 2020 • edited Loading

elasticmachine commented Dec 11, 2020

kevinlog commented Dec 11, 2020

ruflin commented Dec 14, 2020

kevinlog commented Dec 14, 2020

skh left a comment

Choose a reason for hiding this comment

kevinlog commented Dec 14, 2020

ruflin commented Dec 14, 2020

kevinlog commented Dec 14, 2020

ruflin commented Dec 14, 2020

kevinlog commented Dec 14, 2020

kevinlog commented Dec 14, 2020

jonathan-buttner left a comment

Choose a reason for hiding this comment

jonathan-buttner Dec 14, 2020

Choose a reason for hiding this comment

pzl Dec 14, 2020

Choose a reason for hiding this comment

jonathan-buttner Dec 14, 2020

Choose a reason for hiding this comment

kibanamachine commented Dec 14, 2020

💚 Build Succeeded

Metrics [docs]

Async chunks

Distributable file count

Page load bundle

History

nnamdifrankie commented Dec 9, 2020 •

edited by kevinlog

Loading

kevinlog commented Dec 11, 2020 •

edited

Loading