Customizable job interval / managed index interval

[dbbaughe]

Right now every Managed Index job is scheduled to run every 5 minutes by default and can be controlled by the opendistro.index_state_management.job_interval setting for increasing or decreasing.

This is a global setting that all jobs use (with the caveat of changing it doesn't take affect until after the next execution of each job). It is possible though that you might want some jobs to run more frequently than others, i.e. you have some indices that might be more important and should run more frequently to catch rollover/transition conditions faster or execute some actions more frequently that some old indices that is just going to delete itself in 90 days.

To take it even further while inside a single job you might actually want certain states to execute more frequently. The first state of your policy that is waiting to move to warm in 24 hours might execute a lot more frequently than the last state that is waiting to delete the index after a couple months.

We could keep taking this further if there was a need (i.e. actions can have different intervals).

The job would then use whichever job interval is more granular. This could allow a user to free up consumed resources from specific states that really don't need to be executed as frequently; especially when dealing with hundreds to thousands of indices.