Skip to content
This repository has been archived by the owner on Aug 2, 2022. It is now read-only.

Performance: Look into moving ManagedIndexMetaData from cluster state into index #207

Open
dbbaughe opened this issue Apr 24, 2020 · 0 comments
Assignees
Labels
maintenance improves code quality, but not the product

Comments

@dbbaughe
Copy link
Contributor

Every Managed Index job has its own lifecycle which includes updating the cluster state for executions.
Because the cluster state is applied sequentially in a single thread and needs acknowledgement from other nodes for each update once you start managing thousands of indices it becomes a bottleneck.

We have some workarounds to help with this (making steps idempotent, only updating cluster state if actual work will be done, increasing cluster state update timeout), but these are mostly just bandaids on the issue and while they might keep us afloat for a while, we're probably not going to hit the scale we want (10k+ managed indices on a cluster).

We should consider moving the ManagedIndexMetaData out of the cluster state and into the config index itself. This is non trivial mainly because of the automatic migration that needs to happen once someone uses an updated plugin from the old one. We'll need to do some extensive testing around this along with performance testing to see the impact.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
maintenance improves code quality, but not the product
Projects
None yet
Development

No branches or pull requests

2 participants