Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CI] MixedClusterClientYamlTestSuiteIT test {p0=indices.segments/10_basic/basic segments test} failing #82014

Closed
pugnascotia opened this issue Dec 21, 2021 · 5 comments
Assignees
Labels
:Distributed Indexing/Store Issues around managing unopened Lucene indices. If it touches Store.java, this is a likely label. Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. >test-failure Triaged test failures from CI

Comments

@pugnascotia
Copy link
Contributor

Build scan:
https://gradle-enterprise.elastic.co/s/ly44eu3fqw5z4/tests/:qa:mixed-cluster:v7.17.0%23mixedClusterTest/org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT/test%20%7Bp0=indices.segments%2F10_basic%2Fbasic%20segments%20test%7D

Reproduction line:
./gradlew ':qa:mixed-cluster:v7.17.0#mixedClusterTest' -Dtests.class="org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT" -Dtests.method="test {p0=indices.segments/10_basic/basic segments test}" -Dtests.seed=2B8A62B9C9DE5613 -Dtests.bwc=true -Dtests.locale=tr -Dtests.timezone=America/Belize -Druntime.java=17

Applicable branches:
8.0

Reproduces locally?:
Didn't try

Failure history:
https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT&tests.test=test%20%7Bp0%3Dindices.segments/10_basic/basic%20segments%20test%7D

Failure excerpt:

java.lang.AssertionError: Failure at [indices.segments/10_basic:35]: expected [2xx] status code but api [indices.segments] returned [500 Internal Server Error] [{"error":{"root_cause":[{"type":"null_pointer_exception","reason":"Cannot invoke \"java.util.Map.put(Object, Object)\" because \"this.indicesSegments\" is null","stack_trace":"org.elasticsearch.ElasticsearchException$1: Cannot invoke \"java.util.Map.put(Object, Object)\" because \"this.indicesSegments\" is null\n\tat org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:639)\n\tat org.elasticsearch.ElasticsearchException.generateFailureXContent(ElasticsearchException.java:567)\n\tat org.elasticsearch.rest.BytesRestResponse.build(BytesRestResponse.java:140)\n\tat org.elasticsearch.rest.BytesRestResponse.<init>(BytesRestResponse.java:101)\n\tat org.elasticsearch.rest.BytesRestResponse.<init>(BytesRestResponse.java:81)\n\tat org.elasticsearch.rest.action.RestActionListener.onFailure(RestActionListener.java:55)\n\tat org.elasticsearch.rest.action.RestActionListener.onResponse(RestActionListener.java:40)\n\tat org.elasticsearch.rest.action.DispatchingRestToXContentListener.lambda$processResponse$0(DispatchingRestToXContentListener.java:51)\n\tat org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:775)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\nCaused by: java.lang.NullPointerException: Cannot invoke \"java.util.Map.put(Object, Object)\" because \"this.indicesSegments\" is null\n\tat org.elasticsearch.action.admin.indices.segments.IndicesSegmentResponse.getIndices(IndicesSegmentResponse.java:76)\n\tat org.elasticsearch.action.admin.indices.segments.IndicesSegmentResponse.addCustomXContentFields(IndicesSegmentResponse.java:94)\n\tat org.elasticsearch.action.support.broadcast.BroadcastResponse.toXContent(BroadcastResponse.java:150)\n\tat org.elasticsearch.rest.action.DispatchingRestToXContentListener$1.buildResponse(DispatchingRestToXContentListener.java:48)\n\tat org.elasticsearch.rest.action.DispatchingRestToXContentListener$1.buildResponse(DispatchingRestToXContentListener.java:44)\n\tat org.elasticsearch.rest.action.RestBuilderListener.buildResponse(RestBuilderListener.java:27)\n\tat org.elasticsearch.rest.action.RestResponseListener.processResponse(RestResponseListener.java:26)\n\tat org.elasticsearch.rest.action.RestActionListener.onResponse(RestActionListener.java:38)\n\t... 7 more\n\tSuppressed: java.lang.IllegalStateException: Failed to close the XContentBuilder\n\t\tat org.elasticsearch.xcontent.XContentBuilder.close(XContentBuilder.java:1210)\n\t\tat org.elasticsearch.rest.action.RestBuilderListener.buildResponse(RestBuilderListener.java:26)\n\t\t... 9 more\n\tCaused by: java.io.IOException: Unclosed object or array found\n\t\tat org.elasticsearch.xcontent.json.JsonXContentGenerator.close(JsonXContentGenerator.java:456)\n\t\tat org.elasticsearch.xcontent.XContentBuilder.close(XContentBuilder.java:1208)\n\t\t... 10 more\n"}],"type":"null_pointer_exception","reason":"Cannot invoke \"java.util.Map.put(Object, Object)\" because \"this.indicesSegments\" is null","stack_trace":"java.lang.NullPointerException: Cannot invoke \"java.util.Map.put(Object, Object)\" because \"this.indicesSegments\" is null\n\tat org.elasticsearch.action.admin.indices.segments.IndicesSegmentResponse.getIndices(IndicesSegmentResponse.java:76)\n\tat org.elasticsearch.action.admin.indices.segments.IndicesSegmentResponse.addCustomXContentFields(IndicesSegmentResponse.java:94)\n\tat org.elasticsearch.action.support.broadcast.BroadcastResponse.toXContent(BroadcastResponse.java:150)\n\tat org.elasticsearch.rest.action.DispatchingRestToXContentListener$1.buildResponse(DispatchingRestToXContentListener.java:48)\n\tat org.elasticsearch.rest.action.DispatchingRestToXContentListener$1.buildResponse(DispatchingRestToXContentListener.java:44)\n\tat org.elasticsearch.rest.action.RestBuilderListener.buildResponse(RestBuilderListener.java:27)\n\tat org.elasticsearch.rest.action.RestResponseListener.processResponse(RestResponseListener.java:26)\n\tat org.elasticsearch.rest.action.RestActionListener.onResponse(RestActionListener.java:38)\n\tat org.elasticsearch.rest.action.DispatchingRestToXContentListener.lambda$processResponse$0(DispatchingRestToXContentListener.java:51)\n\tat org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:775)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\n\tSuppressed: java.lang.IllegalStateException: Failed to close the XContentBuilder\n\t\tat org.elasticsearch.xcontent.XContentBuilder.close(XContentBuilder.java:1210)\n\t\tat org.elasticsearch.rest.action.RestBuilderListener.buildResponse(RestBuilderListener.java:26)\n\t\t... 9 more\n\tCaused by: java.io.IOException: Unclosed object or array found\n\t\tat org.elasticsearch.xcontent.json.JsonXContentGenerator.close(JsonXContentGenerator.java:456)\n\t\tat org.elasticsearch.xcontent.XContentBuilder.close(XContentBuilder.java:1208)\n\t\t... 10 more\n","suppressed":[{"type":"illegal_state_exception","reason":"Failed to close the XContentBuilder","caused_by":{"type":"i_o_exception","reason":"Unclosed object or array found","stack_trace":"java.io.IOException: Unclosed object or array found\n\tat org.elasticsearch.xcontent.json.JsonXContentGenerator.close(JsonXContentGenerator.java:456)\n\tat org.elasticsearch.xcontent.XContentBuilder.close(XContentBuilder.java:1208)\n\tat org.elasticsearch.rest.action.RestBuilderListener.buildResponse(RestBuilderListener.java:26)\n\tat org.elasticsearch.rest.action.RestResponseListener.processResponse(RestResponseListener.java:26)\n\tat org.elasticsearch.rest.action.RestActionListener.onResponse(RestActionListener.java:38)\n\tat org.elasticsearch.rest.action.DispatchingRestToXContentListener.lambda$processResponse$0(DispatchingRestToXContentListener.java:51)\n\tat org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:775)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\n"},"stack_trace":"java.lang.IllegalStateException: Failed to close the XContentBuilder\n\tat org.elasticsearch.xcontent.XContentBuilder.close(XContentBuilder.java:1210)\n\tat org.elasticsearch.rest.action.RestBuilderListener.buildResponse(RestBuilderListener.java:26)\n\tat org.elasticsearch.rest.action.RestResponseListener.processResponse(RestResponseListener.java:26)\n\tat org.elasticsearch.rest.action.RestActionListener.onResponse(RestActionListener.java:38)\n\tat org.elasticsearch.rest.action.DispatchingRestToXContentListener.lambda$processResponse$0(DispatchingRestToXContentListener.java:51)\n\tat org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:775)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\nCaused by: java.io.IOException: Unclosed object or array found\n\tat org.elasticsearch.xcontent.json.JsonXContentGenerator.close(JsonXContentGenerator.java:456)\n\tat org.elasticsearch.xcontent.XContentBuilder.close(XContentBuilder.java:1208)\n\t... 10 more\n"}]},"status":500}]

  at __randomizedtesting.SeedInfo.seed([2B8A62B9C9DE5613:A3DE5D6367223BEB]:0)
  at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:489)
  at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.test(ESClientYamlSuiteTestCase.java:462)
  at jdk.internal.reflect.GeneratedMethodAccessor19.invoke(null:-1)
  at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:568)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:44)
  at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:375)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:824)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:475)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:375)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:831)
  at java.lang.Thread.run(Thread.java:833)

  Caused by: java.lang.AssertionError: expected [2xx] status code but api [indices.segments] returned [500 Internal Server Error] [{"error":{"root_cause":[{"type":"null_pointer_exception","reason":"Cannot invoke \"java.util.Map.put(Object, Object)\" because \"this.indicesSegments\" is null","stack_trace":"org.elasticsearch.ElasticsearchException$1: Cannot invoke \"java.util.Map.put(Object, Object)\" because \"this.indicesSegments\" is null\n\tat org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:639)\n\tat org.elasticsearch.ElasticsearchException.generateFailureXContent(ElasticsearchException.java:567)\n\tat org.elasticsearch.rest.BytesRestResponse.build(BytesRestResponse.java:140)\n\tat org.elasticsearch.rest.BytesRestResponse.<init>(BytesRestResponse.java:101)\n\tat org.elasticsearch.rest.BytesRestResponse.<init>(BytesRestResponse.java:81)\n\tat org.elasticsearch.rest.action.RestActionListener.onFailure(RestActionListener.java:55)\n\tat org.elasticsearch.rest.action.RestActionListener.onResponse(RestActionListener.java:40)\n\tat org.elasticsearch.rest.action.DispatchingRestToXContentListener.lambda$processResponse$0(DispatchingRestToXContentListener.java:51)\n\tat org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:775)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\nCaused by: java.lang.NullPointerException: Cannot invoke \"java.util.Map.put(Object, Object)\" because \"this.indicesSegments\" is null\n\tat org.elasticsearch.action.admin.indices.segments.IndicesSegmentResponse.getIndices(IndicesSegmentResponse.java:76)\n\tat org.elasticsearch.action.admin.indices.segments.IndicesSegmentResponse.addCustomXContentFields(IndicesSegmentResponse.java:94)\n\tat org.elasticsearch.action.support.broadcast.BroadcastResponse.toXContent(BroadcastResponse.java:150)\n\tat org.elasticsearch.rest.action.DispatchingRestToXContentListener$1.buildResponse(DispatchingRestToXContentListener.java:48)\n\tat org.elasticsearch.rest.action.DispatchingRestToXContentListener$1.buildResponse(DispatchingRestToXContentListener.java:44)\n\tat org.elasticsearch.rest.action.RestBuilderListener.buildResponse(RestBuilderListener.java:27)\n\tat org.elasticsearch.rest.action.RestResponseListener.processResponse(RestResponseListener.java:26)\n\tat org.elasticsearch.rest.action.RestActionListener.onResponse(RestActionListener.java:38)\n\t... 7 more\n\tSuppressed: java.lang.IllegalStateException: Failed to close the XContentBuilder\n\t\tat org.elasticsearch.xcontent.XContentBuilder.close(XContentBuilder.java:1210)\n\t\tat org.elasticsearch.rest.action.RestBuilderListener.buildResponse(RestBuilderListener.java:26)\n\t\t... 9 more\n\tCaused by: java.io.IOException: Unclosed object or array found\n\t\tat org.elasticsearch.xcontent.json.JsonXContentGenerator.close(JsonXContentGenerator.java:456)\n\t\tat org.elasticsearch.xcontent.XContentBuilder.close(XContentBuilder.java:1208)\n\t\t... 10 more\n"}],"type":"null_pointer_exception","reason":"Cannot invoke \"java.util.Map.put(Object, Object)\" because \"this.indicesSegments\" is null","stack_trace":"java.lang.NullPointerException: Cannot invoke \"java.util.Map.put(Object, Object)\" because \"this.indicesSegments\" is null\n\tat org.elasticsearch.action.admin.indices.segments.IndicesSegmentResponse.getIndices(IndicesSegmentResponse.java:76)\n\tat org.elasticsearch.action.admin.indices.segments.IndicesSegmentResponse.addCustomXContentFields(IndicesSegmentResponse.java:94)\n\tat org.elasticsearch.action.support.broadcast.BroadcastResponse.toXContent(BroadcastResponse.java:150)\n\tat org.elasticsearch.rest.action.DispatchingRestToXContentListener$1.buildResponse(DispatchingRestToXContentListener.java:48)\n\tat org.elasticsearch.rest.action.DispatchingRestToXContentListener$1.buildResponse(DispatchingRestToXContentListener.java:44)\n\tat org.elasticsearch.rest.action.RestBuilderListener.buildResponse(RestBuilderListener.java:27)\n\tat org.elasticsearch.rest.action.RestResponseListener.processResponse(RestResponseListener.java:26)\n\tat org.elasticsearch.rest.action.RestActionListener.onResponse(RestActionListener.java:38)\n\tat org.elasticsearch.rest.action.DispatchingRestToXContentListener.lambda$processResponse$0(DispatchingRestToXContentListener.java:51)\n\tat org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:775)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\n\tSuppressed: java.lang.IllegalStateException: Failed to close the XContentBuilder\n\t\tat org.elasticsearch.xcontent.XContentBuilder.close(XContentBuilder.java:1210)\n\t\tat org.elasticsearch.rest.action.RestBuilderListener.buildResponse(RestBuilderListener.java:26)\n\t\t... 9 more\n\tCaused by: java.io.IOException: Unclosed object or array found\n\t\tat org.elasticsearch.xcontent.json.JsonXContentGenerator.close(JsonXContentGenerator.java:456)\n\t\tat org.elasticsearch.xcontent.XContentBuilder.close(XContentBuilder.java:1208)\n\t\t... 10 more\n","suppressed":[{"type":"illegal_state_exception","reason":"Failed to close the XContentBuilder","caused_by":{"type":"i_o_exception","reason":"Unclosed object or array found","stack_trace":"java.io.IOException: Unclosed object or array found\n\tat org.elasticsearch.xcontent.json.JsonXContentGenerator.close(JsonXContentGenerator.java:456)\n\tat org.elasticsearch.xcontent.XContentBuilder.close(XContentBuilder.java:1208)\n\tat org.elasticsearch.rest.action.RestBuilderListener.buildResponse(RestBuilderListener.java:26)\n\tat org.elasticsearch.rest.action.RestResponseListener.processResponse(RestResponseListener.java:26)\n\tat org.elasticsearch.rest.action.RestActionListener.onResponse(RestActionListener.java:38)\n\tat org.elasticsearch.rest.action.DispatchingRestToXContentListener.lambda$processResponse$0(DispatchingRestToXContentListener.java:51)\n\tat org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:775)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\n"},"stack_trace":"java.lang.IllegalStateException: Failed to close the XContentBuilder\n\tat org.elasticsearch.xcontent.XContentBuilder.close(XContentBuilder.java:1210)\n\tat org.elasticsearch.rest.action.RestBuilderListener.buildResponse(RestBuilderListener.java:26)\n\tat org.elasticsearch.rest.action.RestResponseListener.processResponse(RestResponseListener.java:26)\n\tat org.elasticsearch.rest.action.RestActionListener.onResponse(RestActionListener.java:38)\n\tat org.elasticsearch.rest.action.DispatchingRestToXContentListener.lambda$processResponse$0(DispatchingRestToXContentListener.java:51)\n\tat org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:775)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\nCaused by: java.io.IOException: Unclosed object or array found\n\tat org.elasticsearch.xcontent.json.JsonXContentGenerator.close(JsonXContentGenerator.java:456)\n\tat org.elasticsearch.xcontent.XContentBuilder.close(XContentBuilder.java:1208)\n\t... 10 more\n"}]},"status":500}]

    at org.junit.Assert.fail(Assert.java:88)
    at org.elasticsearch.test.rest.yaml.section.DoSection.execute(DoSection.java:374)
    at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:478)
    at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.test(ESClientYamlSuiteTestCase.java:462)
    at jdk.internal.reflect.GeneratedMethodAccessor19.invoke(null:-1)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:568)
    at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:44)
    at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
    at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
    at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
    at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:375)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:824)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:475)
    at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
    at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
    at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
    at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
    at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
    at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
    at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
    at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:375)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:831)
    at java.lang.Thread.run(Thread.java:833)

@pugnascotia pugnascotia added :Distributed Indexing/Store Issues around managing unopened Lucene indices. If it touches Store.java, this is a likely label. >test-failure Triaged test failures from CI labels Dec 21, 2021
@elasticmachine elasticmachine added the Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. label Dec 21, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed (Team:Distributed)

@droberts195
Copy link
Contributor

droberts195 commented Dec 22, 2021

There are 3 tests that are failing very frequently in 8.0:

  1. test {p0=cat.segments/10_basic/Test cat segments using wildcards}
  2. test {p0=cat.segments/10_basic/Test cat segments output}
  3. test {p0=indices.segments/10_basic/basic segments test}

For example https://gradle-enterprise.elastic.co/s/3263wqwswyn3s/tests/overview?outcome=failed on my PR, but I can see several other people have the same problem.

I will mute them in 8.0.

@droberts195
Copy link
Contributor

droberts195 commented Dec 22, 2021

I will mute them in 8.0.

It looks like the problem also exists in master, at least for 2 of the tests: https://gradle-enterprise.elastic.co/s/ofnmel6ib2wpa/tests/overview?outcome=failed

I'll mute all 3 in both branches for consistency - see #82025

droberts195 added a commit to droberts195/elasticsearch that referenced this issue Dec 22, 2021
droberts195 added a commit that referenced this issue Dec 22, 2021
droberts195 added a commit to droberts195/elasticsearch that referenced this issue Dec 22, 2021
@droberts195
Copy link
Contributor

There are more tests failing too, so many that even the backport of the PR to mute the 3 tests mentioned above failed.

I strongly suspect this is being caused by the changes to IndicesSegmentResponse.java in #81556

I am going to try reverting both that change and my muting PR and see what happens.

@arteam
Copy link
Contributor

arteam commented Jan 12, 2022

Fixed via #81556

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Distributed Indexing/Store Issues around managing unopened Lucene indices. If it touches Store.java, this is a likely label. Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. >test-failure Triaged test failures from CI
Projects
None yet
Development

No branches or pull requests

4 participants