Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] [Remote Store] testRefreshSuccessOnThirdAttemptAttempt and RemoteStoreRefreshListenerTests.testRefreshSuccessOnSecondAttempt flaky test failure #8945

Closed
dreamer-89 opened this issue Jul 28, 2023 · 3 comments
Closed

[BUG] [Remote Store] testRefreshSuccessOnThirdAttemptAttempt and RemoteStoreRefreshListenerTests.testRefreshSuccessOnSecondAttempt flaky test failure #8945

dreamer-89 opened this issue Jul 28, 2023 · 3 comments
Assignees
@sachinpkale
Labels
bug Something isn't working distributed framework flaky-test Random test failure that succeeds on second run Indexing:Replication Issues and PRs related to core replication framework eg segrep Storage Issues and PRs relating to data and metadata storage >test-failure Test failure from CI, local build, etc. v2.10.0

Comments

@dreamer-89
Copy link
Member

dreamer-89 commented Jul 28, 2023 •
edited
Loading

Coming from meta issue tracking Segment Replication flaky test failures

8 org.opensearch.index.shard.RemoteStoreRefreshListenerTests.testRefreshSuccessOnThirdAttemptAttempt (19308,19348,19436,19436,19436,19511,19511,19927)
5 org.opensearch.index.shard.RemoteStoreRefreshListenerTests.testRefreshSuccessOnSecondAttempt (19008,19330,19373,19426,19445)
@dreamer-89 dreamer-89 added bug Something isn't working untriaged labels Jul 28, 2023
@dreamer-89 dreamer-89 added >test-failure Test failure from CI, local build, etc. distributed framework flaky-test Random test failure that succeeds on second run Indexing:Replication Issues and PRs related to core replication framework eg segrep Storage Issues and PRs relating to data and metadata storage v2.10.0 and removed untriaged labels Jul 28, 2023
@dreamer-89 dreamer-89 mentioned this issue Jul 28, 2023
Closed
16 tasks
@dreamer-89 dreamer-89 changed the title [BUG] [Remote Store] testRefreshSuccessOnThirdAttemptAttempt flaky test failure [BUG] [Remote Store] testRefreshSuccessOnThirdAttemptAttempt and RemoteStoreRefreshListenerTests.testRefreshSuccessOnSecondAttempt flaky test failure Jul 28, 2023
@sachinpkale sachinpkale self-assigned this Aug 8, 2023
@sachinpkale
Copy link
Member

These tests are already fixed. Ran them on local for 100+ times without failure. Resolving the ticket.

@github-project-automation github-project-automation bot moved this from Todo to Done in Segment Replication Aug 8, 2023
@reta reta reopened this Nov 15, 2023
@github-project-automation github-project-automation bot moved this from Done to In Progress in Segment Replication Nov 15, 2023
@reta
Copy link
Collaborator

reta commented Nov 15, 2023

https://build.ci.opensearch.org/job/gradle-check/30055/testReport/junit/org.opensearch.index.shard/RemoteStoreRefreshListenerTests/testRefreshSuccessOnSecondAttempt/

java.lang.AssertionError: expected:<1> but was:<2>
	at __randomizedtesting.SeedInfo.seed([F53F33D3254DF2EB:E1E6CB6AA722EBA7]:0)
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.failNotEquals(Assert.java:835)
	at org.junit.Assert.assertEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:633)
	at org.opensearch.index.shard.RemoteStoreRefreshListenerTests.lambda$assertNoLagAndTotalUploadsFailed$11(RemoteStoreRefreshListenerTests.java:411)
	at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1089)
	at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1062)
	at org.opensearch.index.shard.RemoteStoreRefreshListenerTests.assertNoLagAndTotalUploadsFailed(RemoteStoreRefreshListenerTests.java:407)
	at org.opensearch.index.shard.RemoteStoreRefreshListenerTests.testRefreshSuccessOnSecondAttempt(RemoteStoreRefreshListenerTests.java:357)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:1583)
	Suppressed: java.lang.AssertionError: expected:<1> but was:<2>
		at org.junit.Assert.fail(Assert.java:89)
		at org.junit.Assert.failNotEquals(Assert.java:835)
		at org.junit.Assert.assertEquals(Assert.java:647)
		at org.junit.Assert.assertEquals(Assert.java:633)
		at org.opensearch.index.shard.RemoteStoreRefreshListenerTests.lambda$assertNoLagAndTotalUploadsFailed$11(RemoteStoreRefreshListenerTests.java:411)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1077)
		... 41 more
	Suppressed: java.lang.AssertionError: expected:<1> but was:<2>
		at org.junit.Assert.fail(Assert.java:89)
		at org.junit.Assert.failNotEquals(Assert.java:835)
		at org.junit.Assert.assertEquals(Assert.java:647)
		at org.junit.Assert.assertEquals(Assert.java:633)
		at org.opensearch.index.shard.RemoteStoreRefreshListenerTests.lambda$assertNoLagAndTotalUploadsFailed$11(RemoteStoreRefreshListenerTests.java:411)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1077)
		... 41 more
	Suppressed: java.lang.AssertionError: expected:<1> but was:<2>
		at org.junit.Assert.fail(Assert.java:89)
		at org.junit.Assert.failNotEquals(Assert.java:835)
		at org.junit.Assert.assertEquals(Assert.java:647)
		at org.junit.Assert.assertEquals(Assert.java:633)
		at org.opensearch.index.shard.RemoteStoreRefreshListenerTests.lambda$assertNoLagAndTotalUploadsFailed$11(RemoteStoreRefreshListenerTests.java:411)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1077)
		... 41 more
	Suppressed: java.lang.AssertionError: expected:<1> but was:<2>
		at org.junit.Assert.fail(Assert.java:89)
		at org.junit.Assert.failNotEquals(Assert.java:835)
		at org.junit.Assert.assertEquals(Assert.java:647)
		at org.junit.Assert.assertEquals(Assert.java:633)
		at org.opensearch.index.shard.RemoteStoreRefreshListenerTests.lambda$assertNoLagAndTotalUploadsFailed$11(RemoteStoreRefreshListenerTests.java:411)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1077)
		... 41 more
	Suppressed: java.lang.AssertionError: expected:<1> but was:<2>
		at org.junit.Assert.fail(Assert.java:89)
		at org.junit.Assert.failNotEquals(Assert.java:835)
		at org.junit.Assert.assertEquals(Assert.java:647)
		at org.junit.Assert.assertEquals(Assert.java:633)
		at org.opensearch.index.shard.RemoteStoreRefreshListenerTests.lambda$assertNoLagAndTotalUploadsFailed$11(RemoteStoreRefreshListenerTests.java:411)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1077)
		... 41 more
	Suppressed: java.lang.AssertionError: expected:<1> but was:<2>
		at org.junit.Assert.fail(Assert.java:89)
		at org.junit.Assert.failNotEquals(Assert.java:835)
		at org.junit.Assert.assertEquals(Assert.java:647)
		at org.junit.Assert.assertEquals(Assert.java:633)
		at org.opensearch.index.shard.RemoteStoreRefreshListenerTests.lambda$assertNoLagAndTotalUploadsFailed$11(RemoteStoreRefreshListenerTests.java:411)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1077)
		... 41 more
	Suppressed: java.lang.AssertionError: expected:<1> but was:<2>
		at org.junit.Assert.fail(Assert.java:89)
		at org.junit.Assert.failNotEquals(Assert.java:835)
		at org.junit.Assert.assertEquals(Assert.java:647)
		at org.junit.Assert.assertEquals(Assert.java:633)
		at org.opensearch.index.shard.RemoteStoreRefreshListenerTests.lambda$assertNoLagAndTotalUploadsFailed$11(RemoteStoreRefreshListenerTests.java:411)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1077)
		... 41 more
	Suppressed: java.lang.AssertionError: expected:<1> but was:<2>
		at org.junit.Assert.fail(Assert.java:89)
		at org.junit.Assert.failNotEquals(Assert.java:835)
		at org.junit.Assert.assertEquals(Assert.java:647)
		at org.junit.Assert.assertEquals(Assert.java:633)
		at org.opensearch.index.shard.RemoteStoreRefreshListenerTests.lambda$assertNoLagAndTotalUploadsFailed$11(RemoteStoreRefreshListenerTests.java:411)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1077)
		... 41 more
	Suppressed: java.lang.AssertionError: expected:<1> but was:<2>
		at org.junit.Assert.fail(Assert.java:89)
		at org.junit.Assert.failNotEquals(Assert.java:835)
		at org.junit.Assert.assertEquals(Assert.java:647)
		at org.junit.Assert.assertEquals(Assert.java:633)
		at org.opensearch.index.shard.RemoteStoreRefreshListenerTests.lambda$assertNoLagAndTotalUploadsFailed$11(RemoteStoreRefreshListenerTests.java:411)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1077)
		... 41 more
	Suppressed: java.lang.AssertionError: expected:<1> but was:<2>
		at org.junit.Assert.fail(Assert.java:89)
		at org.junit.Assert.failNotEquals(Assert.java:835)
		at org.junit.Assert.assertEquals(Assert.java:647)
		at org.junit.Assert.assertEquals(Assert.java:633)
		at org.opensearch.index.shard.RemoteStoreRefreshListenerTests.lambda$assertNoLagAndTotalUploadsFailed$11(RemoteStoreRefreshListenerTests.java:411)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1077)
		... 41 more
	Suppressed: java.lang.AssertionError: expected:<1> but was:<2>
		at org.junit.Assert.fail(Assert.java:89)
		at org.junit.Assert.failNotEquals(Assert.java:835)
		at org.junit.Assert.assertEquals(Assert.java:647)
		at org.junit.Assert.assertEquals(Assert.java:633)
		at org.opensearch.index.shard.RemoteStoreRefreshListenerTests.lambda$assertNoLagAndTotalUploadsFailed$11(RemoteStoreRefreshListenerTests.java:411)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1077)
		... 41 more
	Suppressed: java.lang.AssertionError: expected:<1> but was:<2>
		at org.junit.Assert.fail(Assert.java:89)
		at org.junit.Assert.failNotEquals(Assert.java:835)
		at org.junit.Assert.assertEquals(Assert.java:647)
		at org.junit.Assert.assertEquals(Assert.java:633)
		at org.opensearch.index.shard.RemoteStoreRefreshListenerTests.lambda$assertNoLagAndTotalUploadsFailed$11(RemoteStoreRefreshListenerTests.java:411)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1077)
		... 41 more
	Suppressed: java.lang.AssertionError: expected:<1> but was:<2>
		at org.junit.Assert.fail(Assert.java:89)
		at org.junit.Assert.failNotEquals(Assert.java:835)
		at org.junit.Assert.assertEquals(Assert.java:647)
		at org.junit.Assert.assertEquals(Assert.java:633)
		at org.opensearch.index.shard.RemoteStoreRefreshListenerTests.lambda$assertNoLagAndTotalUploadsFailed$11(RemoteStoreRefreshListenerTests.java:411)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1077)
		... 41 more

@reta reta mentioned this issue Nov 15, 2023
Merged
8 tasks
@Dharin-shah Dharin-shah mentioned this issue Dec 17, 2023
Merged
8 tasks
@andrross andrross mentioned this issue Jan 4, 2024
Closed
@sachinpkale
Copy link
Member

Taking a look

@reta reta mentioned this issue Jan 12, 2024
Merged
@bowenlan-amzn bowenlan-amzn mentioned this issue Jan 16, 2024
Merged
8 tasks
@reta reta mentioned this issue Jan 18, 2024
Merged
@sachinpkale sachinpkale mentioned this issue Feb 3, 2024
Merged
8 tasks
@github-project-automation github-project-automation bot moved this from In Progress to Done in Segment Replication Feb 9, 2024
@kotwanikunal kotwanikunal mentioned this issue Apr 12, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sachinpkale sachinpkale

Labels
bug Something isn't working distributed framework flaky-test Random test failure that succeeds on second run Indexing:Replication Issues and PRs related to core replication framework eg segrep Storage Issues and PRs relating to data and metadata storage >test-failure Test failure from CI, local build, etc. v2.10.0
Projects
Segment Replication
Status: Done
Milestone
No milestone
Development

No branches or pull requests
4 participants
@reta @ankitkala @sachinpkale @dreamer-89