NoSuchElementException: Pool exhausted when starting app

[He-Pin]

Bug Report

Current Behavior

Stack trace

Caused by: java.util.NoSuchElementException: Pool exhausted
  at io.lettuce.core.support.BoundedAsyncPool.acquire()(Unknown Source)

Input Code

Input Code

 client = RedisClient.create();
        client.setOptions(
            ClientOptions.builder()
                .autoReconnect(true)
                .cancelCommandsOnReconnectFailure(true)
                .requestQueueSize(1024 * Runtime.getRuntime().availableProcessors())
                .build());
        final RedisURI redisURI = RedisURI.builder()
            .withHost(redisAddr)
            .withPort(6379) //默认值
            .withPassword(redisPassword)
            .build();

        pool = AsyncConnectionPoolSupport.createBoundedObjectPool(
            () -> client.connectAsync(StringCodec.UTF8, redisURI),
            //默认8个连接
            //refs:https://github.com/lettuce-io/lettuce-core/issues/1363
            BoundedPoolConfig.builder()
                .maxTotal(Runtime.getRuntime().availableProcessors() * 2)
                .minIdle(Runtime.getRuntime().availableProcessors())
                .build());
        final String pong = pool.acquire().get().sync().ping();

Expected behavior/code

Environment

    <lettuce.version>5.3.4.RELEASE</lettuce.version>

Possible Solution

Additional context

[He-Pin]

Isn't this fixed in refs:#1363
?

[mp911de]

I also think that's a duplicate of #1363. By awaiting initialization via BoundedAsyncPool.create(…), the pool should no longer throw exceptions. I also think we should investigate whether we can detect in-flight creations and attach the acquisition to the asynchronous pool warming. Can you upgrade to 5.3.4 and retest whether the issue persists?

[He-Pin]

@mp911de Yes, I was using the 5.3.4, so I was wondering...

I am currently using lettuce to connect to aliyun's redis.
I wrote this code in the @PostConstract method, so I will try to dig this out.

[mp911de]

#1363 introduced two new methods BoundedAsyncPool.create(…) and AsyncConnectionPoolSupport.createBoundedObjectPoolAsync(…) that return CompletionStage<Pool>. We didn't want to introduce blocking behavior so adding new methods was the only option to allow for synchronization.

[He-Pin]

@mp911de I will read through the PR and feedback.

[He-Pin]

@mp911de It's still there after I updated the code

[mp911de]

Thanks for report. We need to check why this is.

[He-Pin]

@mp911de After I set the maxTotal to match the concurrent acquire threads number, it disappears.

[mp911de]

Providing a minimal sample that reproduces the problem would be helpful. Since I don't have any insight into what the application is doing it's not possible to further diagnose the issue.

[He-Pin]

@mp911de I will close this one, cause it's turn out to be an underside issue. thanks.

[vsharathis]

@He-Pin , I encountered same issue, can you please share the code snippet to mitigate the issue ?
Timeout occurred while populating hgetall : java.util.concurrent.CompletionException: java.util.NoSuchElementException: Pool exhausted

[Brico87]

Hi @vsharathis, I do not know if this is the correct way of doing stuff (@mp911de could validate) but to avoid that error, it seems you need to prepare your async pool like that:

public AsyncPool<StatefulRedisConnection<String, String>> createAsyncConnectionPool(RedisURI redisURI, int connectionPoolSize) {
      var boundedPoolConfig = BoundedPoolConfig.builder()
              .minIdle(connectionPoolSize)
              .maxTotal(connectionPoolSize)
              .build();
      return AsyncConnectionPoolSupport.createBoundedObjectPool(() -> redisClient.connectAsync(StringCodec.UTF8, redisURI), boundedPoolConfig);
}

By forcing the pool to initialize all connections in IDLE mode, then no "pool exhauted" error :)

Cheers !

[vsharathis]

@Brico87 , Thanks for the comment, what is the best "connectionPoolSize" to manage, is it based the number of transactions per second or do need to follow some formula to define it. Please do clarify , thanks

[Brico87]

I think I let that to @mp911de answer that but I put a random 10 :)

[vsharathis]

After setting pool size to 1000 and executor thread to 1000 still we encountering pool size exhaust exception @mp911de @He-Pin @Brico87 @jongyeol please share your comments on this topic thanks in advance

[Brico87]

I try to test many solutions on my side and I finish by NOT using the AsyncPool. Instead, I used a GenericObjectPool (using ConnectionPoolSupport.createGenericObjectPool). And then, you can "borrow" connection and you can still do some connection.async and this is working like a charm !

==> https://lettuce.io/core/release/reference/#connection-pooling.blocking.connection-pool-support

Do you absolutely need to use AsyncPool @vsharathis ?

[sksamuel]

I am seeing this issue too - approx 500K requests a minute, and this error occurs around 200 times.

[vsharathis]

@Brico87 , yes ASYC is must , in ConnectionPoolSupport.createGenericObjectPool getting same error
2022/06/17 17:15:53.597 {LettuceWorker-075} [JUnitTest:testGetAllElementsFromMapInCluster] <redis.info-log> (Always) 'redis.info-log' Redis info: readAllFromMap:borrowObject:Pool exhausted
java.util.NoSuchElementException: Pool exhausted
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:451)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:356)
at io.lettuce.core.support.ConnectionPoolSupport$1.borrowObject(ConnectionPoolSupport.java:121)
at io.lettuce.core.support.ConnectionPoolSupport$1.borrowObject(ConnectionPoolSupport.java:117)
at com.lucent.aaa.plugin.Redis$ConnectionContext.readAllFromMap(Redis.java:1061)
at com.lucent.aaa.plugin.Redis.executeSelectAllFrommap(Redis.java:2822)
at com.lucent.aaa.plugin.Redis.executeCommand(Redis.java:2749)
at com.lucent.aaa.plugin.Redis.lambda$invokePlugin$0(Redis.java:1828)
at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)

@Brico87 @He-Pin @mp911de please share some suggestions

[vsharathis]

Hi @sksamuel, possible to share your code on connection and executing commend

[ovkulkarni4]

@vsharathis , can you elaborate the need of async here?

[He-Pin]

IIRC, I was increase the max pooled number, I have no access to the codes now. @vsharathis ，sorry。

[demkom58]

Why would it even throw an exception like this instead of returning the connection later when it's available?