Performance: Look into moving ManagedIndexMetaData from cluster state into index

[adityaj1107]

Issue by dbbaughe
Friday Apr 24, 2020 at 16:15 GMT
Originally opened as opendistro-for-elasticsearch/index-management#207

---

Every Managed Index job has its own lifecycle which includes updating the cluster state for executions.
Because the cluster state is applied sequentially in a single thread and needs acknowledgement from other nodes for each update once you start managing thousands of indices it becomes a bottleneck.

We have some workarounds to help with this (making steps idempotent, only updating cluster state if actual work will be done, increasing cluster state update timeout), but these are mostly just bandaids on the issue and while they might keep us afloat for a while, we're probably not going to hit the scale we want (10k+ managed indices on a cluster).

We should consider moving the ManagedIndexMetaData out of the cluster state and into the config index itself. This is non trivial mainly because of the automatic migration that needs to happen once someone uses an updated plugin from the old one. We'll need to do some extensive testing around this along with performance testing to see the impact.

[dbbaughe]

This was completed by opendistro-for-elasticsearch/index-management#280