[BUG] Gracefully terminate index refresh job when Flint index deleted accidentally

[dai-chen]

What is the bug?

In issue #237, the recover index API has been updated to address the scenario where the metadata log entry is cleaned up if a Flint data index in OpenSearch is accidentally deleted by a user. This change prevents an infinite attempt to recover the Flint index. However, it's essential to note that the original job also requires special handling to ensure it gracefully exits, avoiding the risk of hanging indefinitely.

What is the expected behavior?

The index refresh job should terminate gracefully if Flint data index is deleted accidentally. Need to figure out if this is doable in Spark streaming job internally or it has to be done externally in monitor system.