[BUG] Test org.opensearch.indices.replication.SegmentReplicationIT.testReplicaHasDiffFilesThanPrimary is flaky

[kotwanikunal]

Describe the bug

• org.opensearch.indices.replication.SegmentReplicationIT.testReplicaHasDiffFilesThanPrimary is flaky
• https://build.ci.opensearch.org/job/gradle-check/13054/testReport/org.opensearch.indices.replication/SegmentReplicationIT/testReplicaHasDiffFilesThanPrimary_2/

To Reproduce
Steps to reproduce the behavior:

./gradlew ':server:internalClusterTest' --tests "org.opensearch.indices.replication.SegmentReplicationIT.testReplicaHasDiffFilesThanPrimary" -Dtests.seed=40450506CE0A9D4A -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=es-PE -Dtests.timezone=Antarctica/Vostok -Druntime.java=19

Screenshots

• https://build.ci.opensearch.org/job/gradle-check/13054

Host/Environment (please complete the following information):

• Jenkins Build Server

Additional context
Add any other context about the problem here.

[dreamer-89]

Failing seed

./gradlew ':server:internalClusterTest' --tests "org.opensearch.indices.replication.SegmentReplicationIT.testReplicaHasDiffFilesThanPrimary" -Dtests.seed=40450506CE0A9D4A -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=es-PE -Dtests.timezone=Antarctica/Vostok -Druntime.java=19

org.opensearch.indices.replication.SegmentReplicationIT > testReplicaHasDiffFilesThanPrimary FAILED
    java.lang.AssertionError: Count is 8 hits but 10 was expected.  Total shards: 1 Successful shards: 1 & 0 shard failures:
        at __randomizedtesting.SeedInfo.seed([40450506CE0A9D4A:B7A9A677146A423D]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.opensearch.test.hamcrest.OpenSearchAssertions.assertHitCount(OpenSearchAssertions.java:303)
        at org.opensearch.indices.replication.SegmentReplicationBaseIT.assertDocCounts(SegmentReplicationBaseIT.java:100)
        at org.opensearch.indices.replication.SegmentReplicationIT.lambda$testReplicaHasDiffFilesThanPrimary$6(SegmentReplicationIT.java:747)
        at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1060)
        at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1033)
        at org.opensearch.indices.replication.SegmentReplicationIT.testReplicaHasDiffFilesThanPrimary(SegmentReplicationIT.java:747)

[saratvemulapalli]

Re-opening this: The tests are flaky and hitting into failures again. See: #7302 (comment)

[mch2]

Fixed a portion of this with linked PR which fails when replicas are not sync'd but still getting failures for seed -Dtests.seed=788A6B8A3DB5D6B4.
Now it fails with

五月 11, 2023 7:02:39 上午 com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
警告: Uncaught exception in thread: Thread[#79,opensearch[node_t2][refresh][T#1],5,TGRP-SegmentReplicationIT]
java.lang.AssertionError: afterRefresh called but not beforeRefresh
	at __randomizedtesting.SeedInfo.seed([788A6B8A3DB5D6B4]:0)
	at org.opensearch.index.shard.IndexShard$RefreshMetricUpdater.afterRefresh(IndexShard.java:4315)
	at org.apache.lucene.search.ReferenceManager.notifyRefreshListenersRefreshed(ReferenceManager.java:275)
	at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:182)
	at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:240)
	at org.opensearch.index.engine.NRTReplicationEngine.refresh(NRTReplicationEngine.java:336)
	at org.opensearch.index.shard.IndexShard.refresh(IndexShard.java:1255)
	at org.opensearch.action.admin.indices.refresh.TransportShardRefreshAction.lambda$shardOperationOnReplica$1(TransportShardRefreshAction.java:110)
	at org.opensearch.action.ActionListener.completeWith(ActionListener.java:342)
	at org.opensearch.action.admin.indices.refresh.TransportShardRefreshAction.shardOperationOnReplica(TransportShardRefreshAction.java:109)
	at org.opensearch.action.admin.indices.refresh.TransportShardRefreshAction.shardOperationOnReplica(TransportShardRefreshAction.java:57)
	at org.opensearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.onResponse(TransportReplicationAction.java:758)
	at org.opensearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.onResponse(TransportReplicationAction.java:729)
	at org.opensearch.index.shard.IndexShard.lambda$innerAcquireReplicaOperationPermit$41(IndexShard.java:3873)
	at org.opensearch.action.ActionListener$3.onResponse(ActionListener.java:130)
	at org.opensearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:309)
	at org.opensearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:254)
	at org.opensearch.index.shard.IndexShard.lambda$acquireReplicaOperationPermit$39(IndexShard.java:3801)
	at org.opensearch.index.shard.IndexShard.innerAcquireReplicaOperationPermit(IndexShard.java:3921)
	at org.opensearch.index.shard.IndexShard.acquireReplicaOperationPermit(IndexShard.java:3795)
	at org.opensearch.action.support.replication.TransportReplicationAction.acquireReplicaOperationPermit(TransportReplicationAction.java:1204)
	at org.opensearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.doRun(TransportReplicationAction.java:849)
	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
	at org.opensearch.action.support.replication.TransportReplicationAction.handleReplicaRequest(TransportReplicationAction.java:697)
	at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:106)
	at org.opensearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:453)
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:806)