Bootstrap profiling indices at startup

[danielmitterdorfer]

With this commit we bootstrap all indices and related structures that are required for Elastic Universal Profiling.

[danielmitterdorfer]

This is in preparation of setting up a K/V index scheme that allows the profiling solution to use the out of the box implementation of ILM in Elasticsearch (related to #85273). What's missing?

• Support for versioning upgrades of indices
• Implementation of an index scheme for K/V indices that allows to use an ILM policy for these indices as well

Both will be tackled in follow-up PRs.

[danielmitterdorfer]

@elasticmachine merge upstream

[danielmitterdorfer]

I am a bit uncertain which team would be most appropriate to look at this. I drew inspiration from #57629 when applying labels. Please feel free to relabel if required.

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[elasticsearchmachine]

Hi @danielmitterdorfer, I've created a changelog YAML for you.

[danielmitterdorfer]

@elasticmachine merge upstream

[danielmitterdorfer]

There was an unrelated test failure. I raised #96000.

[danielmitterdorfer]

@elasticmachine run elasticsearch-ci/part-2

[danielmitterdorfer]

I'm gonna look separately into TransformUsageIT#testUsage() but despite that the PR is ready for review now.

[danielmitterdorfer]

I've pushed a preliminary fix for the test failure here (to unblock progress on the PR) and have raised #96015 separately. I'll incorporate #96015 once it's merged to main.

[gmarouli]

Is this policy reflecting the current set-up? If not, what is the rational behind the warm tier and the shrink action? Are they necessary or do they add enough value, for example?

[danielmitterdorfer]

In this PR I've basically only ported the existing policy as it is defined in https://github.com/elastic/kibana/blob/main/x-pack/plugins/profiling/server/lib/setup/steps/ilm_profiling.json but - as discussed offline - I agree that not everything in there makes sense. I'll address this later in a separate PR.

[gmarouli]

LGTM, @danielmitterdorfer thank you for breaking it into steps :). A few minor comments, that's all

[danielmitterdorfer]

Thank you very much for the review @gmarouli!