[BUG] org.opensearch.action.admin.indices.create.RemoteSplitIndexIT is flaky

[reta]

Describe the bug
The whole test suite org.opensearch.action.admin.indices.create.RemoteSplitIndexI seems to be flaky:

org.opensearch.action.admin.indices.create.RemoteSplitIndexIT.testCreateSplitIndex
org.opensearch.action.admin.indices.create.RemoteSplitIndexIT.testCreateSplitIndexToN
org.opensearch.action.admin.indices.create.RemoteSplitIndexIT.testSplitFromOneToN
org.opensearch.action.admin.indices.create.RemoteSplitIndexIT.testSplitIndexPrimaryTerm
org.opensearch.action.admin.indices.create.RemoteSplitIndexIT.classMethod

java.lang.AssertionError: All incoming requests on node [node_s0] should have finished. Expected 0 but got 4580; pending tasks [[{
  "node" : "node_s0",
  "id" : 610,
  "type" : "transport",
  "action" : "internal:index/shard/recovery/start_recovery",
  "description" : "",
  "start_time" : "2023-11-04T02:12:00.316Z",
  "start_time_in_millis" : 1699063920316,
  "running_time" : "4.1m",
  "running_time_in_nanos" : 247039486299,
  "cancellable" : false,
  "cancelled" : false,
  "headers" : { },
  "resource_stats" : {
    "average" : {
      "cpu_time_in_nanos" : 0,
      "memory_in_bytes" : 0
    },
    "total" : {
      "cpu_time_in_nanos" : 0,
      "memory_in_bytes" : 0
    },
    "min" : {
      "cpu_time_in_nanos" : 0,
      "memory_in_bytes" : 0
    },
    "max" : {
      "cpu_time_in_nanos" : 0,
      "memory_in_bytes" : 0
    },
    "thread_info" : {
      "thread_executions" : 0,
      "active_threads" : 0
    }
  }
}]]

nov 03, 2023 7:12:00 PM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
AVVERTENZA: Uncaught exception in thread: Thread[#637,opensearch[node_s0][generic][T#4],5,TGRP-RemoteSplitIndexIT]
java.lang.AssertionError: shard [target][0] is not locked
	at __randomizedtesting.SeedInfo.seed([35E40FF221179300]:0)
	at org.opensearch.env.NodeEnvironment.deleteShardDirectoryUnderLock(NodeEnvironment.java:579)
	at org.opensearch.indices.IndicesService.deleteShardStore(IndicesService.java:1215)
	at org.opensearch.index.IndexService.onShardClose(IndexService.java:675)
	at org.opensearch.index.IndexService$StoreCloseListener.accept(IndexService.java:798)
	at org.opensearch.index.IndexService$StoreCloseListener.accept(IndexService.java:785)
	at org.opensearch.index.store.Store.closeInternal(Store.java:573)
	at org.opensearch.index.store.Store$1.closeInternal(Store.java:193)
	at org.opensearch.common.util.concurrent.AbstractRefCounted.decRef(AbstractRefCounted.java:78)
	at org.opensearch.index.store.Store.decRef(Store.java:546)
	at org.opensearch.action.ActionRunnable$1.doRun(ActionRunnable.java:61)
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:908)
	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)

To Reproduce

 ./gradlew ':server:internalClusterTest' --tests "org.opensearch.action.admin.indices.create.RemoteSplitIndexIT" -Dtests.seed=35E40FF221179300

Expected behavior
The test should always pass

Plugins
Standard

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• CI

Additional context

• https://build.ci.opensearch.org/job/gradle-check/2953

[gbbafna]

This should get fixed by #11720 . Verified by running this in a loop.