Unstopped thread when using Kerberos authentication (with proposed solution)

[HoffmannTom]

Driver version

mssql-jdbc-10.2.0.jre17.jar

SQL Server version

SQL Server 2014

Client Operating System

Win10, Win2016

JAVA/JVM version

jdk17.0.0_35 (Corretto)

Table schema

--

Problem description

We are maintining a Java Web-application, running with Tomcat 10. After switching to Kerberos-Auth, Tomcat reported a memory leak because a Thread was started by the application and never stopped.
I stumbled upon this bug report: #918
This is exactly the cause.
During undeployment of the application, the corresponding classloader gets discharged.
The JDBC-driver spawned a new thread (AddressChangeListener) by doing some DNS lookups.
The spawned AddressChangeListener is losing its contextclassloader when undeployment finished.

More details are mentioned in the issue 918, thats why I wan't do provide a possible solution.
The problem is, that the thread is spawned with the webapp-classloader.
Before doing the DNS-stuff, the context-classloader should be changed.

Code-Example:

    ClassLoader cl = null;
    try
    {
        cl = Thread.currentThread().getContextClassLoader();
        if (ClassLoader.getSystemClassLoader() != null)
               Thread.currentThread().setContextClassLoader(ClassLoader.getSystemClassLoader());
        // do the dangerous DNS stuff
    }
    finally
    {
        Thread.currentThread().setContextClassLoader(cl);
    } 

Would this solution be feasible to you?

Another option would be to report the bug to the JDK-Team. If someone has connections to that team, it usually
helps to speed up a ticket. Was this behaviour reported to the JDK-Team?

Expected behavior

The web-application / jdbc driver shouldn't spawn a thread which can't be stopped.

Actual behavior

Kerberos auth is using DNS lookup which causes a thread to be spawned with the webapps classloader.

Error message/stack trace

WARNING [Thread-9] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [Thread-7] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@17.0.2/sun.net.dns.ResolverConfigurationImpl.notifyAddrChange0(Native Method)
java.base@17.0.2/sun.net.dns.ResolverConfigurationImpl$AddressChangeListener.run(ResolverConfigurationImpl.java:176)

Any other details that can be helpful

--

JDBC trace logs

--

[Jeffery-Wasty]

Hi @HoffmannTom,

Thank you for both the report and the proposed solution. We'll take a look at both and get back to you as soon as we can.

[Jeffery-Wasty]

Hi @HoffmannTom,

After reading #918, this appears to be related to AddressChangeListener once again. Looking at your stack trace, the warning comes about after AddressChangeListener has been run. In #918, the conclusion is that, the thread is created by Java on purpose, and will continue to exist as long as you use Kerberos in Java. If multiple threads were being created, this would indeed be a problem, but that doesn't seem to be the case because of the static nature of the thread.

If, after multiple runs, multiple threads are created OR there indeed seems to be a memory leak (as opposed to a warning), please let us know, as this is something we do not expect, but otherwise we expect this warning to persist, and believe that it should not cause any harm to your application.

You are more than welcome to continue this discussion with the JDK-team, though I would suspect they have been alerted to this issue through issue #918, and have already made a decision. We don't expect there to be any difference in response time between you independently reaching out, and reaching out through our referral.

[HoffmannTom]

Hello Jeffery,
thanks for re-evaluating the topic and for the quick reply.
The AddressChangeListener only runs once. Redeployment doesn't duplicate the thread.

My concern is, that the thread was created with a context-classloader which get's discharged during undeployment.
Thus the thread has a reference to this discharged class-loader which might prevent the JVM to cleanup that classloader.
The spawned thread holds a reference to this classloader: https://docs.oracle.com/javase/7/docs/api/java/lang/Thread.html#getContextClassLoader()

When re-deploying multiple times, the memory leak doesn't grow because the thread is only created once.
But the first classloader which created the thread, won't be cleaned up by the jvm (GC) because of that reference.

To prevent this (small) memory leak, the context-classloader can be switched to the systemclassloader or alternatively
to the parent classloader.

Greetings, Thomas

[Jeffery-Wasty]

Hi @HoffmannTom,

Thank you for your reply. Even though the thread is being created with a context-classloader in this case, due to the fact that there is no leak being created, and there are no other observable problems, this is not an issue we'll be addressing in the short term. Long term we'll definitely be looking at how to manage this situation in a way where we don't have unstopped threads.

Thank you again for your contribution.

[HoffmannTom]

Hallo @Jeffery-Wasty ,
I understand, that this is neither an urgent, nor a high priority ticket.
It causes a small memory leak because the thread class AddressChangeListener inherits a member variable from java.lang.Thread which holds a reference to the classloader of the web-application:

public class Thread implements Runnable {
....
    /* The context ClassLoader for this thread */
    private ClassLoader contextClassLoader;

Thus, when the web-application gets discharged, the corresponding classloader won't get garbage collected because the class AddressChangeListener holds a reference to that classloader.

As the memory leak only happens once (as there is only one AddressChangeListener-Thread) and the leak doesn't grow, it is usually not a big issue.
It is of course up to you, how you want to handle the topic. e.g. if you put it into your backlog or if it is not worth spending efforts in it.

Thanks!
Thomas

[Jeffery-Wasty]

Thank you once again for reporting the issue, and the dialogue you have provided. We have decided to move this issue to our backlog and look at it once again at a later time.