This repository has been archived by the owner on Aug 2, 2022. It is now read-only.
Performance: Look into moving ManagedIndexMetaData from cluster state into index #207
Labels
maintenance
improves code quality, but not the product
Every Managed Index job has its own lifecycle which includes updating the cluster state for executions.
Because the cluster state is applied sequentially in a single thread and needs acknowledgement from other nodes for each update once you start managing thousands of indices it becomes a bottleneck.
We have some workarounds to help with this (making steps idempotent, only updating cluster state if actual work will be done, increasing cluster state update timeout), but these are mostly just bandaids on the issue and while they might keep us afloat for a while, we're probably not going to hit the scale we want (10k+ managed indices on a cluster).
We should consider moving the ManagedIndexMetaData out of the cluster state and into the config index itself. This is non trivial mainly because of the automatic migration that needs to happen once someone uses an updated plugin from the old one. We'll need to do some extensive testing around this along with performance testing to see the impact.
The text was updated successfully, but these errors were encountered: