[BUG] spring-cloud-azure-starter-servicebus-jms - com.azure.core.amqp.exception.AmqpException: onSessionRemoteClose amqp: connection:forced The connection was closed by container because it did not have any active links in the past 300000 milliseconds #41736
Comments
|
Thank you for your feedback. Tagging and routing to the team member best able to assist. |
|
Hello @Tom-Van-Asch, thank you for the report. Since your screenshot shows the breakpoint being hit, are you able to reproduce this locally in your developer machine? If so, could you provide more details
|
|
@Tom-Van-Asch, just to clarify, I mean if the local setup also hits the reported "NPE" (in addition to the |
|
Hi @Tom-Van-Asch. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue. |
|
The same stack trace is discussed, and root caused in this thread #41584 |
|
Thanks for the update, when the new version is released I'll test it again and update this issue with my findings. |
|
We have the same issue without any possible solution to that |
|
the release PR has been opened and will be shipped soon: #42088 |
Thank you Anu for the update. Looking forward to this update. |
|
Hello @Tom-Van-Asch, @pretti-vusion, @amacbean, @josebarros2025, @hylander0, @padmapriyanalam, @pje477, The library update has been released; please follow the steps outlined below. Let us know if the experience improves. Note that you will still see the session disconnect/reconnect logs (which is expected) but the new library should address the NullPointerException. Update to 7.17.4 or 7.17.5 dependency<dependency>
<groupId>com.azure</groupId>
<artifactId>azure-messaging-servicebus</artifactId>
<version>7.17.5</version>
</dependency>Update the ServiceBusClientBuilder for "com.azure.core.amqp.cache"When building any client (ServiceBusProcessorClient, ServiceBusReceiverClient, ServiceBusSenderClient etc..) use the configuration ("com.azure.core.amqp.cache"), as shown below. Make sure this configuration is selected for all the places where the application creates a new new ServiceBusClientBuilder()
.connectionString(CONNECTION_STRING)
.configuration(new ConfigurationBuilder()
.putProperty("com.azure.core.amqp.cache", "true")
.build())
.processor()|sender()|..Choosing this configuration is important to resolve the problem - Ensure right transitive dependenciesMake sure the transitive dependencies (azure-core-amqp, azure-core) are resolved to expected versions. Note: In later versions the need for opt-in "com.azure.core.amqp.cache" will be removed |
|
Hello @pje477, I'm following up on this comment #41584 (comment) you left in the other GitHub issue. While onRemoteSessionClose (normal) and NullPointerException (abnormal) had existed in the versions you listed, the "reactor-executor leak" is something new. Could you please try 7.17.4 steps listed in the above comment and check if it resolves the NPE and associated issues? Few questions about your env where you were observing leak (It's fine to respond later, after trying 7.17.4) -
|
|
I'm having issues like these
this is spring boot 3.3.3 application My dependency tree is different than yours, how should I proceed? |
|
Hi @anuchandy - Thank you for releasing version 7.17.4 of Service Bus SDK - it appears to have fixed our issue! Regarding the reactor-executor thread leak, what we observed is that in versions prior to 7.17.4, when the below error was logged for certain service bus namespaces, the reactor-executor thread associated to that connection would not be closed, and the number of reactor-executor threads would increase over time. This was the error that always preceded the thread leak: We observed this initially as slowly increasing CPU usage of our application, which is a Spring Boot microservice deployed to Azure Spring Apps. We ran a JFR on the Spring Apps instance and observed the high number of zombie reactor-executor threads. Then during troubleshooting we found that only certain service bus namespaces exhibit this behavior. For us, the only namespace that exhibits this behavior is our production instance, and none of our non-prod service bus namespaces exhibit the behavior, despite the SB namespaces being configured identically (via Terraform) and the same exact code being deployed in prod and non-prod. In any case, here are the answers you requested:
String connectionString = "Endpoint=sb://" + hostname + ";SharedAccessKeyName=" + username + ";SharedAccessKey=" + password;
client = new ServiceBusClientBuilder()
.connectionString(connectionString)
.sender()
.queueName(queueTopic)
.buildClient();
client.createMessageBatch();Then wait for 15 minutes. After 15 minutes (exactly), we get the error messages in the attached file. This issue only happens on our production service bus, but for that SB namespace, it happens every time the connection goes idle for 15m or more.
Again, the thread leak behavior appears to be remediated in version 7.17.4 of Service Bus SDK but I'm documenting this here for others. Also - our use case is low-latency message transfer, so we open a connection to a database and another conenction to Service Bus and we keep these connections open for long periods of time. Then when any message is published from the database, it can be transferred to a Service Bus topic as quickly as possible (without the delay of opening the connection to Service Bus). There are periods of time when the message volume drops and there are no messages transferred in a 15-minute window, which is when we were seeing the behavior. |
|
@josebarros2025, thank you for reaching out. I think the issue you're facing is because of spring-cloud-azure-starter:5.16.0 having those core versions (2.9.6, 1.49.1) as the transitive dependency, causing conflicts. The next version of spring-cloud-azure-starter bumping the core version is yet to be released. Have you tried explicitly specifying the required versions of azure-[core|core-amqp] (in addition to azure-messaging-servicebus:7.14.4) in your Spring app POM above the spring-cloud-azure-starter dependency? Typically, in a Console app the first version found in the dependency chain will be used in case of conflict, I'm unsure if Spring dependency resolution behaves in a different way. If you’re unable to override the versions in Spring app, then unfortunately, you'll need to wait for the azure spring team to release the spring-cloud-azure-starter that uses azure-messaging-servicebus:7.14.4. Generally, the azure spring team schedules releases after all the azure SDKs and BOM for that month are available. Given the previous release timelines, I would expect the releases by azure spring team to happen before mid of Oct. |
|
Hello @pje477, that's wonderful news!. Thank you for confirming that 7.14.4 fixes NPE and leak. Also appreciate responding to my questions. It's unfortunate that, like your non-prod namespace, we (SDK Team) were also unable to repro any of this with our test namespaces. Therefore, your assistance in verifying version 7.14.4 was quite valuable. |
|
Hi @anuchandy Thank you for the latest version. I have updated the dependencies to
and updated code to Deployed the changes to demo environment. Still seeing the exception in demo logs com.azure.core.amqp.exception.AmqpException: onSessionRemoteClose connectionId[connectionId], entityName[entityName] condition[Error{condition=amqp:connection:forced, description='The connection was closed by container 'containerId' because it did not have any active links in the past 300000 milliseconds. TrackingId:TrackingId, SystemTracker:gateway10, Timestamp:2024-10-02T12:26:06', info=null}], errorContext[NAMESPACE: servicebusName. ERROR CONTEXT: N/A, PATH: path] |
Hi we aren't having this issue anymore |
|
@josebarros2025, glad to hear you managed to solve the conflicts. Thanks for sharing the solution; I'm sure others will find it helpful. |
|
@padmapriyanalam, thanks for the response. As I noted in my previous comment, we will still observe disconnect events. The service will disconnect if there is no activity, and the client will reconnect during the next send attempt. The log you’re seeing is such a disconnect event. What 7.14.4 addresses is NullPointerException related to the session disconnect event and resulting thread leaks in certain environments. Hope this clarifies. |
|
Closing this issue as 7.17.4 with the fix is released. Refer the steps outlined here to use 7.17.4 |
|
Thank you @josebarros2025 @anuchandy. We aren't having this issue anymore |
|
@anuchandy I believe this issue still persists, even after updating the versions, we are still receiving the error We are using the implementation: spring-cloud-azure-stream-binder-servicebus |
|
Hello @robsonkades, I think the issue is that the configuration "com.azure.core.amqp.cache" is not enabled. Since the application indirectly uses version 7.17.4 through the Spring library and cannot set it directly in the builder, you can set the system property "com.azure.core.amqp.cache" to true. |
|
Hello @anuchandy
We have created shows as below. Request you/team kindly let us know fix for this First we tried by setting App Setting that also didnt solved NullPointerException Azure dependencies are : |
|
Hello @ravibarkhani, it is very likely that the environment configuration "com.azure.core.amqp.cache" was not properly reflected in the JVM, resulting in the code path leading to a NullPointerException being triggered. Could you do the following –
Version 7.17.6 has "com.azure.core.amqp.cache" set to "true" by default hence there is no need to set it explicitly in the builder. So please try updating the version and make sure the |
|
Hi @anuchandy, I'm also facing the similar issue, open one bug #42987 for that. |
After upgrade to 7.17.6 version of service bus we do not see errors in application. May be useful for others below are the dependency in my application
|
|
Hi @anuchandy , I'm probably just being dumb, but I can't get how configuring via configuration builder works in this case. new ServiceBusClientBuilder()
.connectionString(CONNECTION_STRING)
.configuration(new ConfigurationBuilder()
.putProperty("com.azure.core.amqp.cache", "true")
.build())then My dependencies: P.S.: setting "com.azure.core.amqp.cache" as an env variable works just ok, and it fixes the thread leak, so thank you, guys, for the fix anyway. |
|
Has it been noted at all that the same reactor-core thread leak issue happens with Azure Event Hubs as well? I wasn't sure if it's being tracked at all since all the discussion has been around Service Bus. <artifactId>azure-messaging-eventhubs</artifactId>I'm using the latest azure-messaging-eventhubs version as of now (5.19.1) and even added the cache config property as shown below: new EventHubClientBuilder()
.fullyQualifiedNamespace(eventHubNamespaceName)
.eventHubName(eventHubName)
.credential(credential)
.configuration(new ConfigurationBuilder()
.putProperty("com.azure.core.amqp.cache", "true")
.build())
.buildProducerClient();But I continue to see the same behavior. Seeing the following error: and accumulate as many reactor-executor threads (in runnable state) as there are errors.
My Azure SDK Dependencies |
|
Hello @izluben, interesting, that's seems different from what I'm seeing while debugging, which turn picking up "com.azure.core.amqp.cache" configuration object provided in the builder. Let me dig a bit. I am glad to hear that you are unblocked via environment route. |
|
Hi @czoromba1, thanks for reaching out. For Event Hubs, it's a different configuration. Could you try setting "com.azure.messaging.eventhubs.v2" to "true" instead? |
|
Thank you, @anuchandy! I'm happy to report that setting My JVM has been running for 37 hours now, and I don't see any signs of memory or thread leaks. On top of that, there are no occurrences of the following NPE: Is there a new azure-messaging-eventhubs version planned that doesn't require setting this config? |
|
Hi @czoromba1, glad to hear you're unblocked with Event Hubs v2 stack. To answer your question - we currently plan to make the v2 stack the default for Event Hubs in the first quarter of 2025, eliminating the need for applications to opt-in. |
Hi @anuchandy ! I'm facing similar issue as @czoromba1, but I'm using spring boot auto configuration for EventHubProducerClient bean. Can you please suggest me a way to overcome thread leaks? |
|
Hi @Haricshore, thanks for reaching out. I'm not sure if you're using
This should run the To confirm v2 stack is loaded, enable DEBUG log, and confirm we're seeing logs from the two v2 stack classes - If the v2 stack is not enabled for some reason (e.g., |
|
Hi @anuchandy application is failed to send messages to the topic and could see following error in our logs. Error:
|
|
Hello, could you collect -15/+5 minutes of SDK DEBUG logs around the above error log, instructions to collect the logs can be found here (note: "AMQP transport logs" are NOT required but only Azure SDK logs) |
|
Hello @saraswathi-d15, I got access to the logs your shared. During log analysis I see the following log entry - The Could you upgrade the version of spring-cloud-azure-stream-binder-servicebus to 5.19.0 which uses azure-messaging-servicebus version 7.17.7? I don't see any log entries about "Cannot send a message when request response channel is disposed" but let's upgrade as mentioned above and see if that resolves the problem. |
|
I discovered the problem myself in this dependency: <dependency>
<groupId>com.azure</groupId>
<artifactId>azure-messaging-servicebus</artifactId>
<version>7.13.0</version>
</dependency>Upgrading to this one fixed it in my service: <dependency>
<groupId>com.azure</groupId>
<artifactId>azure-messaging-servicebus</artifactId>
<version>7.17.8</version>
</dependency> |
Describe the bug
We are using the ServiceBusSenderClient to send messages to a service bus topic. When there is no message send for 15 minutes we see that the connection is freed but results in een NullPointerException afterwards as a new session is created on the already closed connection.
Exception or Stack Trace
To Reproduce
Code Snippet
Expected behavior
No error logging should occur
Screenshots
When the doFree method is called on the connection the _sessions variable is set to null but afterwards the connection seems to be reused to create a new session which results in the NullPointerException.
Setup (please complete the following information):
The text was updated successfully, but these errors were encountered: