[BUG] BLOB SDK NETTY Exception on Container Creation using Proxy java.lang.ExceptionInInitializerError: null at reactor.netty.resources.PooledConnectionProvider$Pool

[vmaheshw]

Describe the bug
Netty Exception on Container Creation in Azure Blob Storage over proxy.

Exception or Stack Trace

Loaded AZURE_LOG_LEVEL from runtime parameters with value 2.
Loaded java.version from runtime parameters with value 1.8.0_172.
Loaded os.name from runtime parameters with value Linux.

--> PUT https://.blob.core.windows.net/?restype=container
ERROR [] Scheduler worker in group main failed with an uncaught exception
java.lang.ExceptionInInitializerError: null
at reactor.netty.resources.PooledConnectionProvider$Pool.(PooledConnectionProvider.java:227) ~[reactor-netty-0.8.3.RELEASE.jar:0.8.3.RELEASE]
at reactor.netty.resources.PooledConnectionProvider.lambda$acquire$2(PooledConnectionProvider.java:146) ~[reactor-netty-0.8.3.RELEASE.jar:0.8.3.RELEASE]
at reactor.core.publisher.MonoCreate.subscribe(MonoCreate.java:53) ~[reactor-core-3.2.3.RELEASE.jar:3.2.3.RELEASE]
at reactor.netty.http.client.HttpClientConnect$MonoHttpConnect.lambda$subscribe$0(HttpClientConnect.java:318) ~[reactor-netty-0.8.3.RELEASE.jar:0.8.3.RELEASE]
at reactor.core.publisher.MonoCreate.subscribe(MonoCreate.java:53) ~[reactor-core-3.2.3.RELEASE.jar:3.2.3.RELEASE]
at reactor.core.publisher.Mono.subscribe(Mono.java:3608) ~[reactor-core-3.2.3.RELEASE.jar:3.2.3.RELEASE]
at reactor.core.publisher.FluxRetryPredicate$RetryPredicateSubscriber.resubscribe(FluxRetryPredicate.java:123) ~[reactor-core-3.2.3.RELEASE.jar:3.2.3.RELEASE]
at reactor.core.publisher.MonoRetryPredicate.subscribe(MonoRetryPredicate.java:51) ~[reactor-core-3.2.3.RELEASE.jar:3.2.3.RELEASE]
at reactor.netty.http.client.HttpClientConnect$MonoHttpConnect.subscribe(HttpClientConnect.java:321) ~[reactor-netty-0.8.3.RELEASE.jar:0.8.3.RELEASE]
at reactor.core.publisher.MonoFlatMapMany.subscribe(MonoFlatMapMany.java:49) ~[reactor-core-3.2.3.RELEASE.jar:3.2.3.RELEASE]
at reactor.core.publisher.MonoSingle.subscribe(MonoSingle.java:58) ~[reactor-core-3.2.3.RELEASE.jar:3.2.3.RELEASE]
at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.drain(MonoIgnoreThen.java:153) ~[reactor-core-3.2.3.RELEASE.jar:3.2.3.RELEASE]
at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:56) ~[reactor-core-3.2.3.RELEASE.jar:3.2.3.RELEASE]
at reactor.core.publisher.MonoFlatMap.subscribe(MonoFlatMap.java:60) ~[reactor-core-3.2.3.RELEASE.jar:3.2.3.RELEASE]
at reactor.core.publisher.MonoPeekFuseable.subscribe(MonoPeekFuseable.java:74) ~[reactor-core-3.2.3.RELEASE.jar:3.2.3.RELEASE]
at reactor.core.publisher.MonoTimeout.subscribe(MonoTimeout.java:84) ~[reactor-core-3.2.3.RELEASE.jar:3.2.3.RELEASE]
at reactor.core.publisher.MonoDelaySubscription.accept(MonoDelaySubscription.java:49) ~[reactor-core-3.2.3.RELEASE.jar:3.2.3.RELEASE]
at reactor.core.publisher.MonoDelaySubscription.accept(MonoDelaySubscription.java:32) ~[reactor-core-3.2.3.RELEASE.jar:3.2.3.RELEASE]
at reactor.core.publisher.FluxDelaySubscription$DelaySubscriptionOtherSubscriber.onNext(FluxDelaySubscription.java:122) ~[reactor-core-3.2.3.RELEASE.jar:3.2.3.RELEASE]
at reactor.core.publisher.MonoDelay$MonoDelayRunnable.run(MonoDelay.java:117) ~[reactor-core-3.2.3.RELEASE.jar:3.2.3.RELEASE]
at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:50) [reactor-core-3.2.3.RELEASE.jar:3.2.3.RELEASE]
at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:27) [reactor-core-3.2.3.RELEASE.jar:3.2.3.RELEASE]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_172]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_172]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_172]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_172]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_172]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
Caused by: java.lang.IllegalArgumentException: 'channelPool' is already in use
at io.netty.util.ConstantPool.createOrThrow(ConstantPool.java:113) ~[netty-all-4.1.34.Final.jar:4.1.34.Final]
at io.netty.util.ConstantPool.newInstance(ConstantPool.java:95) ~[netty-all-4.1.34.Final.jar:4.1.34.Final]
at io.netty.util.AttributeKey.newInstance(AttributeKey.java:55) ~[netty-all-4.1.34.Final.jar:4.1.34.Final]
at io.netty.channel.pool.SimpleChannelPool.(SimpleChannelPool.java:43) ~[netty-all-4.1.34.Final.jar:4.1.34.Final]
... 28 more

To Reproduce
Azure Container Creation using proxy.

Code Snippet
BlobServiceAsyncClient storageClient1 =
new BlobServiceClientBuilder()
.endpoint(endpoint)
.credential(credential)
.httpClient(ht)
.httpLogDetailLevel(HttpLogDetailLevel.BASIC)
.buildAsyncClient();

Expected behavior
A clear and concise description of what you expected to happen.
No Crash.

Screenshots
If applicable, add screenshots to help explain your problem.

Setup (please complete the following information):

• OS: [e.g. iOS]
• IDE : [e.g. IntelliJ]
• Version of the Library used

Additional context
Add any other context about the problem here.

Information Checklist
Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

• Bug Description Added
• Repro Steps Added
• Setup information Added

[rickle-msft]

Hi, @vmaheshw. Thanks for reporting this.

I notice this line at the bottom of your stack trace:
Caused by: java.lang.IllegalArgumentException: 'channelPool' is already in use

And I also notice that you are specifying your own httpClient that looks to be stored in a variable somewhere. Is it possible that you are sharing the same httpClient across multiple storage clients, which may be what is causing netty to complain that the channel pool is already in use.

[vmaheshw]

No, I'm not sharing my httpClient. It is a local variable in my object and I have created only one object. I'm using httpClient object to pass proxy information. Do you think all the versions of library are correct?

[rickle-msft]

@alzimmermsft @jianghaolu @anuchandy have you ever seen this exception coming from netty before?

[rickle-msft]

@vmaheshw can you share a bit more of your code? Nothing you've shared interacts with Netty at all, so a bit more context in the code would be helpful

[rickle-msft]

Quick update. Alan has suggested offline that somehow this line is getting executed twice:
private static final AttributeKey<SimpleChannelPool> POOL_KEY = AttributeKey.newInstance("channelPool");

[vmaheshw]

Thanks @rickle-msft . Yes, I also suspect that. I'm investigating with the internal team where I see a conflict. I will update you once it is resolved.

[kurtzeborn]

Assigning to @jianghaolu since this appears to be Azure.Core related.

CC: @alzimmermsft

[rickle-msft]

@vmaheshw you mentioned you would check with your internal team. Did anything come of that investigation?

[vmaheshw]

It is fixed as part of FIX in Netty Library.

[ghost]

Thanks for working with Microsoft on GitHub! Tell us how you feel about your experience using the reactions on this comment.