java.lang.LinkageError occured when lookup a remote EJB with multiple threads

[11rx4f]

Environment Details

GlassFish Version (and build number): Eclipse GlassFish 6.2.1 (build master-b577-g50d68a3 2021-09-01T02:07:52+0000)
JDK version: OpenJDK 11.0.12+7
OS: Red Hat Enterprise Linux release 8.0
Database:

Problem Description

java.lang.LinkageError occured when a remote EJB are lookuped with multiple threads.

Caused by: java.lang.LinkageError: loader 'app' attempted duplicate class definition for test.ejb._EJBRemote_Wrapper. (test.ejb._EJBRemote_Wrapper is in unnamed module of loader 'app')
        at java.base/java.lang.ClassLoader.defineClass1(Native Method)
        at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1017)
        ... 19 more

Payara server seems to have fixed this problem as
payara/Payara#3087

Steps to reproduce

1. deploy EJB app
2. lookup the remote EJB with multiple threads

Impact of Issue

Cannot lookup remote EJB with multiple threads.

Additional info to reproduce this issue

LinkageError.zip

$ glassfish6/glassfish/bin/asadmin deploy LinkageError.jar
Application deployed with name LinkageError.
Command deploy executed successfully.
$ java -classpath glassfish6/glassfish/modules/glassfish-naming.jar:glassfish6/glassfish/lib/gf-client.jar:./LinkageError.jar test.client.SessionClient localhost 23700 5 10 0
START : MainClient
START : SessionClient
START : SessionClient
START : SessionClient
START : SessionClient
START : SessionClient
Sep 15, 2021 2:39:36 PM com.sun.enterprise.v3.server.CommonClassLoaderServiceImpl findDerbyClient
INFO: Cannot find javadb client jar file, derby jdbc driver will not be available by default.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.glassfish.pfl.basic.reflection.Bridge (file:/glassfish6/glassfish/modules/pfl-basic.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of org.glassfish.pfl.basic.reflection.Bridge
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Exception in thread "Thread-0" java.lang.Error: Could not access ClassLoader.defineClass()
        at org.glassfish.pfl.basic.reflection.Bridge.defineClass(Bridge.java:106)
        at org.glassfish.pfl.dynamic.codegen.impl.CodeGeneratorUtil.makeClass(CodeGeneratorUtil.java:46)
        at org.glassfish.pfl.dynamic.codegen.spi.Wrapper._generate(Wrapper.java:1026)
        at org.glassfish.pfl.dynamic.codegen.spi.Wrapper._generate(Wrapper.java:1010)
        at com.sun.ejb.EJBUtils.generateAndLoad(EJBUtils.java:569)
        at com.sun.ejb.EJBUtils.loadGeneratedRemoteBusinessClasses(EJBUtils.java:483)
        at com.sun.ejb.EJBUtils.loadGeneratedRemoteBusinessClasses(EJBUtils.java:441)
        at com.sun.ejb.EJBUtils.lookupRemote30BusinessObject(EJBUtils.java:389)
        at com.sun.ejb.containers.RemoteBusinessObjectFactory.getObjectInstance(RemoteBusinessObjectFactory.java:51)
        at java.naming/javax.naming.spi.NamingManager.getObjectInstance(NamingManager.java:341)
        at com.sun.enterprise.naming.impl.SerialContext.getObjectInstance(SerialContext.java:503)
        at com.sun.enterprise.naming.impl.SerialContext.lookup(SerialContext.java:463)
        at com.sun.enterprise.naming.impl.SerialContext.lookup(SerialContext.java:414)
        at java.naming/javax.naming.InitialContext.lookup(InitialContext.java:409)
        at test.client.SessionClient.run(SessionClient.java:77)
Caused by: java.lang.reflect.InvocationTargetException
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.glassfish.pfl.basic.reflection.Bridge.defineClass(Bridge.java:104)
        ... 14 more
Caused by: java.lang.LinkageError: loader 'app' attempted duplicate interface definition for test.ejb._EJBRemote_Remote. (test.ejb._EJBRemote_Remote is in unnamed module of loader 'app')
        at java.base/java.lang.ClassLoader.defineClass1(Native Method)
        at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1017)
        ... 19 more

[dmatej]

I wrote a JMH test and reproduced it even on fixed version.
It seems it is not a race condition, but it is caused by repeating of the action - the PFL generates new class and remembers it.
Then if this class is not remembered also by the classloader, consecutive call tries to generate it again and fails.

I'm not sure yet, but my JMH test reproduces it reliably even in the first iteration. But when I called that just in the unit test, it passed. Bad there is not much logging and eclipse debugger wasn't able to trace it.

[dmatej]

I reproduced it in unit test also just by two consecutive calls of the method, but there was yet problem on another place - some generators bend the API and results are inconsistent; the class was generated, but cannot be loaded.
The JMH test now verified that with another custom generator just for the test the fix merged in recent PR really fixes possible race condition.
I will write yet tests for remaining usages and at least document it. Or make the issue impossible.

[dmatej]

So ... tests found another issue, I will close this and create a new one. This issue was about race condition which was fixed in the PR.