deadlock in orient db 2.2.6

[jamieb22]

orient db 2.2.6
Linux 3.19.0-58-generic #64~14.04.1-Ubuntu SMP Fri Mar 18 19:05:43 UTC 2016 x86_64 x86_64 x86_64
"receive-3" #118 prio=5 os_prio=0 tid=0x00000000146dc800 nid=0x31c6 waiting on condition [0x00007f132a8c9000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000066dc3d038> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
at com.orientechnologies.common.concur.lock.OOneEntryPerKeyLockManager.acquireLock(OOneEntryPerKeyLockManager.java:171)
at com.orientechnologies.common.concur.lock.OOneEntryPerKeyLockManager.acquireLock(OOneEntryPerKeyLockManager.java:100)
at com.orientechnologies.orient.core.storage.impl.local.paginated.atomicoperations.OAtomicOperationsManager.acquireReadLock(OAtomicOperationsManager.java:477)
at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster.readRecord(OPaginatedCluster.java:628)
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.doReadRecord(OAbstractPaginatedStorage.java:3346)
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.readRecord(OAbstractPaginatedStorage.java:2989)
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.readRecord(OAbstractPaginatedStorage.java:1064)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx$SimpleRecordReader.readRecord(ODatabaseDocumentTx.java:3219)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.executeReadRecord(ODatabaseDocumentTx.java:1875)
at com.orientechnologies.orient.core.tx.OTransactionOptimistic.loadRecord(OTransactionOptimistic.java:220)
at com.orientechnologies.orient.core.tx.OTransactionOptimistic.loadRecord(OTransactionOptimistic.java:194)
at com.orientechnologies.orient.core.tx.OTransactionOptimistic.loadRecord(OTransactionOptimistic.java:324)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.load(ODatabaseDocumentTx.java:1615)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.load(ODatabaseDocumentTx.java:101)
at com.orientechnologies.orient.core.id.ORecordId.getRecord(ORecordId.java:330)
at com.tinkerpop.blueprints.impls.orient.OrientElementIterator.next(OrientElementIterator.java:62)
at com.tinkerpop.blueprints.impls.orient.OrientElementIterator.next(OrientElementIterator.java:35)
at com.stimulus.archiva.hm.f(MailArchiva:383)
at com.stimulus.archiva.hm.d(MailArchiva:367)
at com.stimulus.archiva.hm.c(MailArchiva:594)
at com.stimulus.archiva.ns.a(MailArchiva:94)
at com.stimulus.archiva.plugin.blob.MimeMessageBlob.getMemberOfX(MailArchiva:312)
at sun.reflect.GeneratedMethodAccessor94.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.commons.jxpath.util.ValueUtils.getValue(ValueUtils.java:367)
at org.apache.commons.jxpath.ri.model.beans.BeanPropertyPointer.getBaseValue(BeanPropertyPointer.java:120)
at org.apache.commons.jxpath.ri.model.beans.BeanPropertyPointer.getImmediateNode(BeanPropertyPointer.java:149)
at org.apache.commons.jxpath.ri.model.beans.PropertyPointer.getImmediateValuePointer(PropertyPointer.java:161)
at org.apache.commons.jxpath.ri.model.NodePointer.getValuePointer(NodePointer.java:297)
at org.apache.commons.jxpath.ri.JXPathContextReferenceImpl.getValue(JXPathContextReferenceImpl.java:370)
at org.apache.commons.jxpath.ri.JXPathContextReferenceImpl.getValue(JXPathContextReferenceImpl.java:313)
at com.stimulus.archiva.dx.d(MailArchiva:365)
at com.stimulus.archiva.eb.b(MailArchiva:126)
at com.stimulus.archiva.jg.e(MailArchiva:237)
at com.stimulus.archiva.ea.a(MailArchiva:357)
at com.stimulus.archiva.receive.ReceiveService.a(MailArchiva:335)
at com.stimulus.archiva.receive.ReceiveService.a(MailArchiva:147)
at com.stimulus.archiva.receive.ISPReceiveService.a(MailArchiva:80)
at com.stimulus.archiva.receive.ReceiveService.a(MailArchiva:614)
- locked <0x00000006908d5808> (a java.util.LinkedList)
at com.stimulus.archiva.receive.ReceiveService.a(MailArchiva:661)
at com.stimulus.archiva.receive.ReceiveService.a(MailArchiva:395)
at com.stimulus.archiva.queue.d.a(MailArchiva:170)
at com.stimulus.archiva.queue.d.a(MailArchiva:162)
at com.stimulus.archiva.queue.b.b(MailArchiva:270)
at com.stimulus.archiva.queue.b.a(MailArchiva:169)
at com.stimulus.archiva.queue.c$c.run(MailArchiva:192)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"receive-2" #116 prio=5 os_prio=0 tid=0x000000000d38b000 nid=0x31c5 waiting on condition [0x00007f132d7ad000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000066dc3d038> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
at com.orientechnologies.common.concur.lock.OOneEntryPerKeyLockManager.acquireLock(OOneEntryPerKeyLockManager.java:171)
at com.orientechnologies.common.concur.lock.OOneEntryPerKeyLockManager.acquireLock(OOneEntryPerKeyLockManager.java:100)
at com.orientechnologies.orient.core.storage.impl.local.paginated.atomicoperations.OAtomicOperationsManager.acquireReadLock(OAtomicOperationsManager.java:477)
at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster.readRecord(OPaginatedCluster.java:628)
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.doReadRecord(OAbstractPaginatedStorage.java:3346)
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.readRecord(OAbstractPaginatedStorage.java:2989)
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.readRecord(OAbstractPaginatedStorage.java:1064)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx$SimpleRecordReader.readRecord(ODatabaseDocumentTx.java:3219)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.executeReadRecord(ODatabaseDocumentTx.java:1875)
at com.orientechnologies.orient.core.tx.OTransactionOptimistic.loadRecord(OTransactionOptimistic.java:220)
at com.orientechnologies.orient.core.tx.OTransactionOptimistic.loadRecord(OTransactionOptimistic.java:194)
at com.orientechnologies.orient.core.tx.OTransactionOptimistic.loadRecord(OTransactionOptimistic.java:324)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.load(ODatabaseDocumentTx.java:1615)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.load(ODatabaseDocumentTx.java:101)
at com.orientechnologies.orient.core.id.ORecordId.getRecord(ORecordId.java:330)
at com.tinkerpop.blueprints.impls.orient.OrientElementIterator.next(OrientElementIterator.java:62)
at com.tinkerpop.blueprints.impls.orient.OrientElementIterator.next(OrientElementIterator.java:35)
at com.stimulus.archiva.hm.f(MailArchiva:383)
at com.stimulus.archiva.hm.d(MailArchiva:367)
at com.stimulus.archiva.hm.c(MailArchiva:594)
at com.stimulus.archiva.ns.a(MailArchiva:94)
at com.stimulus.archiva.plugin.blob.MimeMessageBlob.getMemberOfX(MailArchiva:312)
at sun.reflect.GeneratedMethodAccessor94.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.commons.jxpath.util.ValueUtils.getValue(ValueUtils.java:367)
at org.apache.commons.jxpath.ri.model.beans.BeanPropertyPointer.getBaseValue(BeanPropertyPointer.java:120)
at org.apache.commons.jxpath.ri.model.beans.BeanPropertyPointer.getImmediateNode(BeanPropertyPointer.java:149)
at org.apache.commons.jxpath.ri.model.beans.PropertyPointer.getImmediateValuePointer(PropertyPointer.java:161)
at org.apache.commons.jxpath.ri.model.NodePointer.getValuePointer(NodePointer.java:297)
at org.apache.commons.jxpath.ri.JXPathContextReferenceImpl.getValue(JXPathContextReferenceImpl.java:370)
at org.apache.commons.jxpath.ri.JXPathContextReferenceImpl.getValue(JXPathContextReferenceImpl.java:313)
at com.stimulus.archiva.dx.d(MailArchiva:365)
at com.stimulus.archiva.eb.b(MailArchiva:126)
at com.stimulus.archiva.jg.e(MailArchiva:237)
at com.stimulus.archiva.ea.a(MailArchiva:357)
at com.stimulus.archiva.receive.ReceiveService.a(MailArchiva:335)
at com.stimulus.archiva.receive.ReceiveService.a(MailArchiva:147)
at com.stimulus.archiva.receive.ISPReceiveService.a(MailArchiva:80)
at com.stimulus.archiva.receive.ReceiveService.a(MailArchiva:614)
- locked <0x00000006908d4168> (a java.util.LinkedList)
at com.stimulus.archiva.receive.ReceiveService.a(MailArchiva:661)
at com.stimulus.archiva.receive.ReceiveService.a(MailArchiva:395)
at com.stimulus.archiva.queue.d.a(MailArchiva:170)
at com.stimulus.archiva.queue.d.a(MailArchiva:162)
at com.stimulus.archiva.queue.b.b(MailArchiva:270)
at com.stimulus.archiva.queue.b.a(MailArchiva:169)
at com.stimulus.archiva.queue.c$c.run(MailArchiva:192)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

catalina.zip

[andrii0lomakin]

Hi @jamieb22 , I looked at your stack traces that is not a deadlock in classical form. All threads are stacked on read lock, and there is no write lock which locks them. The most likely scenarios is following:
on the low level of storage, an exception is thrown as result operation was rolled back but locks keep acquired. I suppose that is related to out of direct memory exception which you got.
What is the situation right now when we applied settings, do you still have this lock issue? Do you see any memory exceptions ?

[jamieb22]

Andrey. You are correct. Orient seems have to stabilized with -XX:MaxDirectMemorySize=512g and restricting the cache to 32k. The default direct memory settings of Orient may be a little too aggressive.

[andrii0lomakin]

Cool . Then I close issues .

пн, 15 авг. 2016 г., 14:10 jamieb22 notifications@github.com:

Andrey. You are correct. Orient seems have to stabilized with
-XX:MaxDirectMemorySize=512g and restricting the cache to 32k. The default
direct memory settings of Orient may be a little too aggressive.

—
You are receiving this because you commented.

Reply to this email directly, view it on GitHub
#6548 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAGaamZ3PiGp1al4HiPOf_PNKrY3vM5Sks5qgEk3gaJpZM4JiKkN
.

Best regards,
Andrey Lomakin, R&D lead.
OrientDB Ltd

twitter: @Andrey_Lomakin
linkedin: https://ua.linkedin.com/in/andreylomakin
blogger: http://andreylomakin.blogspot.com/

[jamieb22]

Andrey ... can you ask your Orient team to make the default direct memory requirement a little less aggressive in future? Also, please can you fix the old index issue with v2.2.

[andrii0lomakin]

Hi @jamieb22 we can not change memory allocation policy by a single request, such approach is usually superb for servers. But if several users of embedded databases will support (or issue) such request we will reconsider it for embedded database.