[BUG] Flaky test SegmentReplicationAllocationIT.testPerIndexPrimaryAllocation

[raghuvanshraj]

Describe the bug

java.lang.AssertionError
	at __randomizedtesting.SeedInfo.seed([9631830FBDB9647:8D5956FBA87A66E1]:0)
	at org.junit.Assert.fail(Assert.java:87)
	at org.junit.Assert.assertTrue(Assert.java:42)
	at org.junit.Assert.assertTrue(Assert.java:53)
	at org.opensearch.indices.replication.SegmentReplicationAllocationIT.lambda$verifyPerIndexPrimaryBalance$0(SegmentReplicationAllocationIT.java:252)

To Reproduce

• Reproducible using the gradle seed
• Build Failure: https://build.ci.opensearch.org/job/gradle-check/16211/testReport/junit/org.opensearch.indices.replication/SegmentReplicationAllocationIT/testPerIndexPrimaryAllocation/

Expected behavior
Test should always pass

[nknize]

Offending assertion is here

assertTrue(primaryCount <= avgPrimaryShardsPerNode);

Where: final int avgPrimaryShardsPerNode = (int) Math.ceil(totalPrimaryShards * 1f / currentState.getRoutingNodes().size());

Does it make sense comparing to an average? Averages are subject to a variance so the actual could occasionally be higher. Seems periodic failures should be expected if an occasional test run produces even one more than the average?