[CI] FullClusterRestartIT.TestRollupAfterRestart failing

[craigtaverner]

Many tests are failing repeatedly over the last couple of days. All have the same errors:

java.lang.RuntimeException: An error occurred orchestrating test cluster.
Caused by: java.io.IOException: (No message provided)
at org.elasticsearch.test.cluster.util.IOUtils.deleteWithRetry0(IOUtils.java:162)

Other tests failing:

• FullClusterRestartIT » testRollupAfterRestart {cluster=UPGRADED}
• MlConfigIndexMappingsFullClusterRestartIT » testMlConfigIndexMappingsAfterM...
• FullClusterRestartIT » testNodeShutdown {cluster=UPGRADED}
• WatcherMappingUpdateIT » testMappingsAreUpdated {cluster=UPGRADED}
• CoreFullClusterRestartIT » testForbidDisableSoftDeletesOnRestore {cluster=U...
• MLModelDeploymentFullClusterRestartIT » testDeploymentSurvivesRestart {clus...
• MlHiddenIndicesFullClusterRestartIT » testMlIndicesBecomeHidden {cluster=UP...
• QueryBuilderBWCIT » testQueryBuilderBWC {p0=UPGRADED}
• FullClusterRestartIT » testSlmPolicyAndStats {cluster=UPGRADED}
• MlConfigIndexMappingsFullClusterRestartIT » testMlConfigIndexMappingsAfterM...
• FullClusterRestartIT » testNodeShutdown {cluster=UPGRADED}
• WatcherMappingUpdateIT » testMappingsAreUpdated {cluster=UPGRADED}
• CoreFullClusterRestartIT » testShrink {cluster=UPGRADED}
• MLModelDeploymentFullClusterRestartIT » testDeploymentSurvivesRestart {clus...
• MlHiddenIndicesFullClusterRestartIT » testMlIndicesBecomeHidden {cluster=UP...
• QueryBuilderBWCIT » testQueryBuilderBWC {p0=UPGRADED}
• and many more...

CI Link

https://gradle-enterprise.elastic.co/s/owosjezvuspf6/tests/:x-pack:qa:full-cluster-restart:v7.17.9%23bwcTest/org.elasticsearch.xpack.restart.FullClusterRestartIT/testRollupAfterRestart%20%7Bcluster=UPGRADED%7D

Repro line

./gradlew ':x-pack:qa:full-cluster-restart:v7.17.9#bwcTest' -Dtests.class="org.elasticsearch.xpack.restart.FullClusterRestartIT" -Dtests.method="testRollupAfterRestart {cluster=UPGRADED}" -Dtests.seed=642657C8CB419BB0 -Dtests.bwc=true -Dtests.locale=es-SV -Dtests.timezone=America/Indiana/Indianapolis -Druntime.java=19

Does it reproduce?

Yes

Applicable branches

main

Failure history

https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.xpack.restart.FullClusterRestartIT&tests.test=testRollupAfterRestart%20%7Bcluster%3DUPGRADED%7D

Failure excerpt

java.lang.RuntimeException: An error occurred orchestrating test cluster.
Caused by: java.io.IOException: (No message provided)
at org.elasticsearch.test.cluster.util.IOUtils.deleteWithRetry0(IOUtils.java:162)

[elasticsearchmachine]

Pinging @elastic/es-delivery (Team:Delivery)

[craigtaverner]

Reverting the PR #93062 allows the test to pass locally, so this appears to be the culprit for these failures.

[valeriy42]

I have noticed several cases where only MlConfigIndexMappingsFullClusterRestartIT was failing: #93504. Maybe this is related.

[mark-vieira]

This was resolved by #93486.