SearchableSnapshotsIntegTests.testCreateAndRestoreSearchableSnapshot failing

[benwtrent]

CI Link

https://gradle-enterprise.elastic.co/s/ivesuuq5fvjji

Repro line

./gradlew ':x-pack:plugin:searchable-snapshots:internalClusterTest' --tests "org.elasticsearch.xpack.searchablesnapshots.SearchableSnapshotsIntegTests.testCreateAndRestoreSearchableSnapshot" -Dtests.seed=613FCFFF93F38F06 -Dtests.locale=es-MX -Dtests.timezone=Africa/Lagos -Druntime.java=22

Does it reproduce?

Not reliably

Applicable branches

lucene_snapshot_9_11

Failure history

No response

Failure excerpt

java.lang.AssertionError: cannot complete future on thread Thread[#199,elasticsearch[node_s2][generic][T#3],5,TGRP-SearchableSnapshotsIntegTests] with waiter on thread Thread[#316,elasticsearch[node_s2][generic][T#11],5,TGRP-SearchableSnapshotsIntegTests], could deadlock if pool was full
--
  | at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:269)
  | at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:756)
  | at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1126)
  | at org.elasticsearch.action.support.PlainActionFuture$Sync.get(PlainActionFuture.java:262)
  | at org.elasticsearch.action.support.PlainActionFuture.get(PlainActionFuture.java:74)
  | at org.elasticsearch.common.util.concurrent.FutureUtils.get(FutureUtils.java:65)
  | at org.elasticsearch.action.support.PlainActionFuture.actionGet(PlainActionFuture.java:171)
  | at org.elasticsearch.xpack.searchablesnapshots.cache.blob.BlobStoreCacheService.get(BlobStoreCacheService.java:126)
  | at org.elasticsearch.xpack.searchablesnapshots.store.SearchableSnapshotDirectory.getCachedBlob(SearchableSnapshotDirectory.java:673)
  | at org.elasticsearch.xpack.searchablesnapshots.store.input.MetadataCachingIndexInput.readWithBlobCache(MetadataCachingIndexInput.java:280)
  | at org.elasticsearch.xpack.searchablesnapshots.store.input.MetadataCachingIndexInput.readInternal(MetadataCachingIndexInput.java:615)
  | at org.elasticsearch.blobcache.common.BlobCacheBufferedIndexInput.refill(BlobCacheBufferedIndexInput.java:304)
  | at org.elasticsearch.blobcache.common.BlobCacheBufferedIndexInput.readByte(BlobCacheBufferedIndexInput.java:57)
  | at org.apache.lucene.store.BufferedChecksumIndexInput.readByte(BufferedChecksumIndexInput.java:39)
  | at org.apache.lucene.codecs.CodecUtil.readBEInt(CodecUtil.java:668)
  | at org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:185)
  | at org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:254)
  | at org.apache.lucene.codecs.lucene94.Lucene94FieldInfosFormat.read(Lucene94FieldInfosFormat.java:138)
  | at org.apache.lucene.index.IndexWriter.readFieldInfos(IndexWriter.java:1259)
  | at org.apache.lucene.index.IndexWriter.getFieldNumberMap(IndexWriter.java:1278)
  | at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1125)
  | at org.elasticsearch.index.store.Store.newTemporaryAppendingIndexWriter(Store.java:1546)
  | at org.elasticsearch.index.store.Store.associateIndexWithNewTranslog(Store.java:1452)
  | at org.elasticsearch.index.shard.StoreRecovery.bootstrap(StoreRecovery.java:620)
  | at org.elasticsearch.index.shard.StoreRecovery.lambda$restore$17(StoreRecovery.java:586)
  | at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)
  | at org.elasticsearch.action.support.SubscribableListener$SuccessResult.complete(SubscribableListener.java:382)
  | at org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:302)
  | at org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:205)
  | at org.elasticsearch.action.support.SubscribableListener.lambda$andThen$0(SubscribableListener.java:469)
  | at org.elasticsearch.action.ActionListener.run(ActionListener.java:433)
  | at org.elasticsearch.action.support.SubscribableListener.newForked(SubscribableListener.java:128)
  | at org.elasticsearch.action.support.SubscribableListener.andThen(SubscribableListener.java:469)
  | at org.elasticsearch.action.support.SubscribableListener.andThen(SubscribableListener.java:433)
  | at org.elasticsearch.index.shard.StoreRecovery.restore(StoreRecovery.java:583)
  | at org.elasticsearch.index.shard.StoreRecovery.recoverFromRepository(StoreRecovery.java:312)
  | at org.elasticsearch.index.shard.IndexShard.restoreFromRepository(IndexShard.java:2481)
  | at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$32(IndexShard.java:3268)
  | at org.elasticsearch.action.ActionRunnable$4.doRun(ActionRunnable.java:100)
  | at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984)
  | at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
  | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
  | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
  | at java.base/java.lang.Thread.run(Thread.java:1570)

[elasticsearchmachine]

Pinging @elastic/es-distributed (Team:Distributed)