java.lang.NoClassDefFoundError: com/hazelcast/util/ExceptionUtil #6701

fredfortier · 2016-09-16T16:55:54Z

OrientDB Version, operating system, or hardware.

v2.1.20

Operating System

Windows Server 2012
WAS 8.5

Expected behavior and actual behavior

I created an error condition assigning two nodes to the same "database" local directory. While I expected an error, what I followed was unexpected. The exception was thrown every few milliseconds in an indefinite loop. Part of the problem seems to be that my hazelcast library does not include the required Exception util.

[9/15/16 16:50:05:058 EDT] 00000a1a ClassicOperat E com.hazelcast.spi.impl.operationexecutor.classic.ClassicOperationExecutor [localhost]:2435 [orientdb] [3.5.3] Failed to process packet: Packet{header=17, isResponse=false, isOperation=true, isEvent=false, partitionId=10, conn=Connection [/127.0.0.1:54033 -> localhost/127.0.0.1:2434], endpoint=Address[localhost]:2434, live=true, type=MEMBER} on hz._hzInstance_1_orientdb.partition-operation.thread-0
java.lang.NoClassDefFoundError: com/hazelcast/util/ExceptionUtil
at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:321)
at com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.processPacket(OperationThread.java:142)
at com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.process(OperationThread.java:115)
at com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.doRun(OperationThread.java:101)
at com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.run(OperationThread.java:76)

[9/15/16 16:50:05:058 EDT] 00000a1b ClassicOperat E com.hazelcast.spi.impl.operationexecutor.classic.ClassicOperationExecutor [localhost]:2435 [orientdb] [3.5.3] Failed to process packet: Packet{header=17, isResponse=false, isOperation=true, isEvent=false, partitionId=11, conn=Connection [/127.0.0.1:54033 -> localhost/127.0.0.1:2434], endpoint=Address[localhost]:2434, live=true, type=MEMBER} on hz._hzInstance_1_orientdb.partition-operation.thread-1
java.lang.NoClassDefFoundError: com/hazelcast/util/ExceptionUtil
at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:321)
at com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.processPacket(OperationThread.java:142)
at com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.process(OperationThread.java:115)
at com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.doRun(OperationThread.java:101)
at com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.run(OperationThread.java:76)

Steps to reproduce the problem

I am running a distributed database embedded in my Java EE application. I am importing the libraries using Maven using the dependencies listed below. Looking at the dependency graph, I expect all hazelcast dependency to come with "orientdb-distributed".

    <dependency>
        <groupId>com.orientechnologies</groupId>
        <artifactId>orientdb-server</artifactId>
        <version>2.1.20</version>
        <scope>compile</scope>
    </dependency>
    <dependency>
        <groupId>com.orientechnologies</groupId>
        <artifactId>orientdb-object</artifactId>
        <version>2.1.20</version>
        <scope>compile</scope>
    </dependency>
    <dependency>
        <groupId>com.orientechnologies</groupId>
        <artifactId>orientdb-enterprise</artifactId>
        <version>2.1.20</version>
        <scope>compile</scope>
    </dependency>
    <dependency>
        <groupId>com.orientechnologies</groupId>
        <artifactId>orientdb-distributed</artifactId>
        <version>2.1.20</version>
        <scope>compile</scope>
    </dependency>

The text was updated successfully, but these errors were encountered:

wolf4ood · 2016-09-19T09:10:47Z

@fredfortier

ExceptionUtil should be in hazelcast jar. That is strange.
Can you check your classpath and see if you have duplicates jar of Hazelcast?

fredfortier · 2016-09-19T17:18:55Z

I see only one hazelcast jar in my classpath: hazelcast-all-3.5.3.jar

I do see a com/hazelcast/util/ExceptionUtil.class in the JAR file.

I am attaching a log excerpt so that you can better understand the context of the error.

hazelcast-error.txt

lvca · 2016-09-19T19:05:24Z

The problem seems to be in the classpath: please assure you have hazelcast-all-3.5.3.jar in your classpath.

fredfortier · 2016-09-19T19:09:36Z

Yes, it's in the classpath for sure. This is a WAR file. The hazelcast jar has the exact same characteristics as all other libraries in the application.

lvca · 2016-09-20T04:07:17Z

If you open the Hazelcast jar, can you find the missing class?

fredfortier · 2016-09-20T14:15:44Z

Yes

I do see a com/hazelcast/util/ExceptionUtil.class in the hazelcast-all-3.5.3.jar file.

wolf4ood · 2016-09-20T14:18:04Z

@fredfortier

so that should means that or you don't have hazelcast-all-3.5.3.jar in the classpath when running your app or actually you have it twice.

fredfortier · 2016-09-20T15:23:33Z

I suspected that WebSphere may ship with a version of hazelcast. However, it does not appear to be the case either. Also, my application uses the "parent last" class loading strategy which would use the application libraries when there is a conflict.

I have also searched application jars for references to hazelcast:
find . -name '*.jar' -exec grep -Hls hazelcast {} \; ./hazelcast-all-3.5.3.jar ./orientdb-distributed-2.1.20.jar

I cleared WebSphere "class cache" and tried to reproduce the issue again. The first error is what I would expect.
Caused by: com.orientechnologies.common.concur.lock.OLockException: File 'C:\IBM\kwcn\orientdb\databases\kwcn\database.ocf' is locked by another process, maybe the database is in use by another process. Use the remote mode with a OrientDB server to allow multiple access to the same database
at com.orientechnologies.orient.core.storage.fs.OFileClassic.lock(OFileClassic.java:674)
at com.orientechnologies.orient.core.storage.fs.OFileClassic.openChannel(OFileClassic.java:731)
at com.orientechnologies.orient.core.storage.fs.OFileClassic.open(OFileClassic.java:523)
at com.orientechnologies.orient.core.storage.impl.local.OSingleFileSegment.open(OSingleFileSegment.java:51)
at com.orientechnologies.orient.core.storage.impl.local.OStorageConfigurationSegment.load(OStorageConfigurationSegment.java:64)
... 132 more

But then it throws the attached Hazelcast exception stack.
hazelcast-error-node2-strike2.txt

Also, although my testing has been light, database replication seems to work just fine otherwise. I can run two instance of my application on separate JVMs, configure their ips in hazelcast.xml and they synchronize beautifully.

fredfortier · 2016-09-20T15:53:52Z

Note that this event also affected my other active node. I had to restart both JVMs to get the database running again.

Caused by:
com.orientechnologies.orient.server.distributed.ODistributedException: Error on executing distributed request (id=-1 from=node1 task=record_delete(#247:72 delayed=false) user=#5:0) against database 'kwcn.[results_node1]' to nodes [node1]
at com.orientechnologies.orient.server.hazelcast.OHazelcastDistributedDatabase.send2Nodes(OHazelcastDistributedDatabase.java:241)
at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.sendRequest(OHazelcastPlugin.java:358)
at com.orientechnologies.orient.server.distributed.ODistributedStorage.deleteRecord(ODistributedStorage.java:815)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.executeDeleteRecord(ODatabaseDocumentTx.java:2101)
at com.orientechnologies.orient.core.tx.OTransactionNoTx.deleteRecord(OTransactionNoTx.java:189)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.delete(ODatabaseDocumentTx.java:2555)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.delete(ODatabaseDocumentTx.java:122)
at com.orientechnologies.orient.core.record.ORecordAbstract.delete(ORecordAbstract.java:296)
at com.enchoice.kwcn.UserConfigService.processResultSet(UserConfigService.java:263)
... 99 more
Caused by:
java.lang.NoClassDefFoundError: com.hazelcast.quorum.QuorumException
at com.hazelcast.spi.Operation.logError(Operation.java:401)
at com.hazelcast.spi.impl.operationservice.impl.operations.Backup.logError(Backup.java:156)
at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.handleOperationError(OperationRunnerImpl.java:265)
at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:141)
at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:315)
at com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.processPacket(OperationThread.java:142)
at com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.process(OperationThread.java:115)
at com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.doRun(OperationThread.java:101)
at com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.run(OperationThread.java:76)
Caused by:
java.lang.ClassNotFoundException: com.hazelcast.quorum.QuorumException
at java.net.URLClassLoader.findClass(URLClassLoader.java:434)
at com.ibm.ws.bootstrap.ExtClassLoader.findClass(ExtClassLoader.java:230)
at java.lang.ClassLoader.loadClassHelper(ClassLoader.java:703)
at java.lang.ClassLoader.loadClass(ClassLoader.java:682)
at com.ibm.ws.bootstrap.ExtClassLoader.loadClass(ExtClassLoader.java:123)
at java.lang.ClassLoader.loadClass(ClassLoader.java:665)
at com.ibm.ws.classloader.ProtectionClassLoader.loadClass(ProtectionClassLoader.java:62)
at com.ibm.ws.classloader.ProtectionClassLoader.loadClass(ProtectionClassLoader.java:58)
at com.ibm.ws.classloader.CompoundClassLoader.loadClass(CompoundClassLoader.java:616)
at java.lang.ClassLoader.loadClass(ClassLoader.java:665)
... 9 more

lvca · 2016-09-20T16:59:45Z

You have to restart the OrientDB node after that change. However, if a class that exists cannot be found, it's not related to OrientDB, but to your container and your configuration. To prove it, please run OrientDB in standalone mode to see if works.

lvca · 2016-09-22T19:04:35Z

Any news on this?

fredfortier · 2016-09-22T19:28:17Z

It's not of an issue with the hazelcast jar not being in the classpath for sure. I verified that. Other than this particular exception condition, data replication works just fine. I have also verified that it is not duplicated in the classpath.

I did not try to validate the same condition in "standalone mode". However, I am confident that it won't be an issue under such perfect conditions.

Unfortunately, my conditions are different from the "standalone mode". My application runs inside an EJB container which means different classloaders. The hazelcast jar is in the classpath of the ejb classloader.

I suspect that hazelcast may be performing some asynchronous method invocation under those particular conditions which might result in a callback under a different classloader scope. I did not have time to test this hypothesis further.

lvca · 2017-06-06T03:34:21Z

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

java.lang.NoClassDefFoundError: com/hazelcast/util/ExceptionUtil #6701

java.lang.NoClassDefFoundError: com/hazelcast/util/ExceptionUtil #6701

fredfortier commented Sep 16, 2016

wolf4ood commented Sep 19, 2016

fredfortier commented Sep 19, 2016

lvca commented Sep 19, 2016

fredfortier commented Sep 19, 2016

lvca commented Sep 20, 2016

fredfortier commented Sep 20, 2016

wolf4ood commented Sep 20, 2016

fredfortier commented Sep 20, 2016

fredfortier commented Sep 20, 2016

lvca commented Sep 20, 2016

lvca commented Sep 22, 2016

fredfortier commented Sep 22, 2016 •

edited

Loading

lvca commented Jun 6, 2017

java.lang.NoClassDefFoundError: com/hazelcast/util/ExceptionUtil #6701

java.lang.NoClassDefFoundError: com/hazelcast/util/ExceptionUtil #6701

Comments

fredfortier commented Sep 16, 2016

OrientDB Version, operating system, or hardware.

Operating System

Expected behavior and actual behavior

Steps to reproduce the problem

wolf4ood commented Sep 19, 2016

fredfortier commented Sep 19, 2016

lvca commented Sep 19, 2016

fredfortier commented Sep 19, 2016

lvca commented Sep 20, 2016

fredfortier commented Sep 20, 2016

wolf4ood commented Sep 20, 2016

fredfortier commented Sep 20, 2016

fredfortier commented Sep 20, 2016

lvca commented Sep 20, 2016

lvca commented Sep 22, 2016

fredfortier commented Sep 22, 2016 • edited Loading

lvca commented Jun 6, 2017

fredfortier commented Sep 22, 2016 •

edited

Loading