[BUG] Shard replication tasks do not start on new nodes intermittently

[soosinha]

What is the bug?
When a node is replace with another node and the shard migrates to the new node, the shard replication task does not start sometimes which results in replication not working.

The bug is in this code where the index task checks for shard tasks for all indices. Even if one shard task is present for any other index it bails out. This code should be modified to check the shard tasks for the particular index only.

How can one reproduce the bug?
Steps to reproduce the behavior:

1. Start replication for atleast 2 indices
2. Add a new node to the cluster
3. Shutdown ES on the node where the index replication task is running
4. Index replication task should start on any of the new nodes
5. Shard replication task may not start on the new node(May have to repeat to this simulate the issue as it is intermittent)

What is the expected behavior?
Index and Shard replication task should start after they are stopped due to node replacement

[krishna-ggk]

Thanks for finding the bug and root-causing it.

One thing to re-evaluate is whether we can unify code-path to start missing shard tasks.

1. When shard tasks are missing for all shards. (link)
2. When shard tasks are missing for few shards. (link)

[ankitkala]

I've raised the PR for fixing the bug where we weren't filtering the ShardReplicationTasks for current index: #497

We should still keep this issue open to refactor and unify the logic to spawn the missing/all shard tasks

[ankitkala]

Closing the issue as we've also addressed the refactoring as part of same PR.