Getting ODistributedRecordLockedException creating edge in distributed mode #5012

stefomak · 2015-09-24T15:27:17Z

Hi, I have orientdb 2.1-SNAPSHOT
running with the -cluster option, but I have only one node running (to make it simpler).
When my java application tries to commit a transaction that involves creations of vertices and edges I get:

Error on transaction commit
com.orientechnologies.orient.server.distributed.task.ODistributedRecordLockedException
at com.orientechnologies.orient.server.distributed.task.OTxTask.execute(OTxTask.java:102)
at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.executeOnLocalNode(OHazelcastPlugin.java:805)
at com.orientechnologies.orient.server.hazelcast.ODistributedWorker.onMessage(ODistributedWorker.java:296)
at com.orientechnologies.orient.server.hazelcast.ODistributedWorker.run(ODistributedWorker.java:119)

In the last three days I tried a lot of combination in order to isolate the problem.
I found out that if the transaction creates only vertices (no edges!!) it doesn't fire any exception.

Do you know any workaround, please?
I'm getting crazy...............

stefomak · 2015-09-24T15:28:35Z

....Thank you!

lvca · 2015-10-07T06:50:52Z

ODistributedRecordLockedException is raised when a concurrent operation is executed on that record. is this your case? What level of concurrency do you have?

In case of TX, it' automatically repeated up to 10 times (by default, setting "distributed.concurrentTxMaxAutoRetry"), so to receive that message seems you have high contention on the same vertices. Can you confirm?

stefomak · 2015-10-07T08:31:18Z

Hi Luca,
my java application tries to commit a transaction that involves creations of vertices and edges: vertex A and other stuff.
I found out that I do something else in a hook.
In the afterCreate method I create a new vertex B and an edge between A e B.
Then I discovered that I need to set DISTRIBUTED_EXECUTION_MODE.
I tried the three options, but only TARGET_NODE works without exceptions.
But doing so, and in distributed mode, I get a new vertex B in the first node of the cluster with a rid different from the rid of the new vertex B in the second node of the cluster.
This solutions doesn't work fine because in this scenario it's impossible to delete the vertex. I got another exception about quorum...
I guess the right DISTRIBUTED_EXECUTION_MODE should be BOTH, but it doesn't work.

Thanks
Stefano

stefomak · 2015-10-16T15:11:56Z

Hi, any idea?

jack18jack · 2016-03-08T12:59:25Z

@stefomak @lvca i got this error too.i try to use spark-orientdb-connector to access distributed orientdb 2.1.12 with two nodes.i raise a question in their github page.do u guys have any findings?
https://github.com/metreta/spark-orientdb-connector/issues/4

nishantkumar1292 · 2017-06-09T10:49:47Z

I got this error too. And as @lvca pointed out that this error is raised when a concurrent operation is executed on that record. I have Vertex A connected to Vertex B1 and Vertex B2 with the same edge class.
My objective is to delete edge between both and I am doing this in loop. First this is executed.
DELETE EDGE FROM #13:438 TO #12:65377
Works fine. Then when executing this
DELETE EDGE FROM #13:438 TO #12:65375
gives error:
com.orientechnologies.orient.server.distributed.task.ODistributedRecordLockedException: Timeout (3756ms) on acquiring lock on record #13:438 on server 'node2'. It is locked by request 0.10791....

Any idea how to solve this?

lvca · 2017-06-09T12:36:43Z

@nishantkumar1292 what release?

nishantkumar1292 · 2017-06-09T16:44:23Z

I am running two server nodes in distributed mode in OrientDB 2.2.17 .

lvca · 2017-06-10T10:13:08Z

Please use v2.2.21.

nishantkumar1292 · 2017-06-13T08:40:32Z

Started using v2.2.21 but now I am not able to start the server nodes. Every node starts as master and the members list does not show the other server. The distributed mode was running fine in 2.2.17. I have the same hazelcast.xml file.
Am I missing something?
Error:
2017-06-13 08:21:49:839 INFO OrientDB Server is active v2.2.21 (build 491997a502b8b90210b269824755c56981765034). [OServer]
2017-06-13 08:23:51:712 INFO [172.87.45.743]:2434 [orientdb] [3.6.5] Could not connect to:/172.31.75.96:2435. Reason: SocketException[Connection timed out to address /172.31.75.96:2435] [InitConnectionTask]
2017-06-13 08:23:51:712 INFO [172.87.45.743]:2434 [orientdb] [3.6.5] Could not connect to: /[ip]:2434. Reason: SocketException[Connection timed out to address /[172.31.75.96]:2434] [InitConnectionTask]
...and this continues. The same is happening for the other server as well.

lvca · 2017-06-13T16:40:51Z

Please check the network configuration in config/hazelcast.xml and config/orientdb-server-config.xml

nishantkumar1292 · 2017-06-22T09:16:37Z

The servers and connections are working fine. However I am still getting the ODistributedRecordLockedException error.

Error: com.orientechnologies.orient.server.distributed.task.ODistributedRecordLockedException: Timeout (5000ms) on acquiring lock on record #12:84870 on server 'node1'. It is locked by request 0.90132.
DB name="db_development"
DB name="db_development"

I am running REPAIR DATABASE --fix-graph -skipEdges=5673222 command in console.

Any idea why is it happening?

lvca · 2017-06-22T13:24:52Z

@nishantkumar1292 Was that exception given during REPAIR DATABASE?

nishantkumar1292 · 2017-06-23T06:36:15Z

yes

roots-ai · 2017-06-30T09:30:20Z

+1 @nishantkumar1292
@ivca! We plan to use orientdb, in production, where there would be
~10,000 requests/sec.
~ 1M Nodes
~ 6M Edges
We couldn't find any way to resolve ODistributedRecordLockedException error while even working with only 2 parallel requests on a node, unsure how this is going to scale up.

Is this a bug with Orientdb 2.2.21, or any other way to get this resolved?

lvca · 2017-06-30T18:05:20Z

Hi guys,
OrientDB has an optimistic approach avoiding locks as much as possible. Only in this way you can scale up massively. In the case there is a concurrent write operation on the same vertices, the infamous ODistributedRecordLockedException is thrown. It's not a bug, this exception extends ONeedRetryException that means it has to be managed on the client side.

All the client code should be like:

for( int retry = 0; retry < 10; ++retry ){
 try {
  // EXECUTE YOUR COMMAND
  break;

 } catch( ONeedRetryException e ){
   // CONTINUE
 }
}

In this way any concurrent operation will be retried (in this case up to 10 times).

nishantkumar1292 · 2017-07-03T08:47:00Z

We are getting this error when executing delete command over an edge class DELETE EDGE edge_class_name through OrientDB studio.
Error: com.orientechnologies.orient.server.distributed.task.ODistributedRecordLockedException: Timeout (592ms) on acquiring lock on record #13:3 on server 'node1'. It is locked by request 0.26 DB name=db_development
Even 10 minutes minutes after this error, we are not able to execute update command on #13:3.
Firstly, how do we solve this?
Secondly, is there a way to figure out which request is locking the node as the error mentions request 0.26. How do we know what is request 0.26?

roots-ai · 2017-07-10T04:31:37Z

@lvca Can you please reply?

lvca · 2017-07-10T05:12:14Z

@roots-ai can you execute a kill -5 <pid> to get from the system output the internal status of the server?

nishantkumar1292 · 2017-07-10T13:43:44Z

The command kill -5 <pid> gave below output.

2017-07-10 12:47:07:741 WARNI Received signal: SIGTRAP [OSignalHandler]
OrientDB 2.2.21 (build 491997a502b8b90210b269824755c56981765034) configuration dump:
- ENVIRONMENT
  + environment.dumpCfgAtStartup = false
  + environment.concurrent = true
  + environment.lockManager.concurrency.level = 64
  + environment.allowJVMShutdown = true
- SCRIPT
  + script.pool.maxSize = 20
- MEMORY
  + memory.useUnsafe = true
  + memory.chunk.size = 2147483647
  + memory.directMemory.safeMode = true
  + memory.directMemory.trackMode = false
  + memory.directMemory.onlyAlignedMemoryAccess = true
- JVM
  + jvm.gc.delayForOptimize = 600
- STORAGE
  + storage.openFiles.limit = 512
  + storage.componentsLock.cache = 10000
  + storage.diskCache.pinnedPages = 20
  + storage.diskCache.bufferSize = 26200
  + storage.diskCache.writeCachePart = 15
  + storage.diskCache.writeCachePageTTL = 86400
  + storage.diskCache.writeCachePageFlushInterval = 25
  + storage.diskCache.writeCacheFlushInactivityInterval = 60000
  + storage.diskCache.writeCacheFlushLockTimeout = -1
  + storage.diskCache.diskFreeSpaceCheckInterval = 5
  + storage.diskCache.diskFreeSpaceCheckIntervalInPages = 2048
  + storage.diskCache.keepState = true
  + storage.diskCache.checksumMode = Store
  + storage.configuration.syncOnUpdate = true
  + storage.compressionMethod = nothing
  + storage.encryptionMethod = nothing
  + storage.encryptionKey = <hidden>
  + storage.makeFullCheckpointAfterCreate = false
  + storage.makeFullCheckpointAfterOpen = true
  + storage.makeFullCheckpointAfterClusterCreate = true
  + storage.trackChangedRecordsInWAL = true
  + storage.useWAL = true
  + storage.wal.syncOnPageFlush = true
  + storage.wal.cacheSize = 3000
  + storage.wal.fileAutoCloseInterval = 10
  + storage.wal.maxSegmentSize = 128
  + storage.wal.maxSize = 4096
  + storage.wal.commitTimeout = 1000
  + storage.wal.shutdownTimeout = 10000
  + storage.wal.fuzzyCheckpointInterval = 300
  + storage.wal.reportAfterOperationsDuringRestore = 10000
  + storage.wal.restore.batchSize = 50000
  + storage.wal.readCacheSize = 1000
  + storage.wal.fuzzyCheckpointShutdownWait = 600
  + storage.wal.fullCheckpointShutdownTimeout = 600
  + storage.wal.path = null
  + storage.diskCache.pageSize = 64
  + storage.diskCache.diskFreeSpaceLimit = 256
  + storage.lowestFreeListBound = 16
  + storage.lockTimeout = 0
  + storage.record.lockTimeout = 2000
  + storage.useTombstones = false
- RECORD
  + record.downsizing.enabled = true
- OBJECT
  + object.saveOnlyDirty = false
- DB
  + db.pool.min = 1
  + db.pool.max = 50
  + db.pool.idleTimeout = 0
  + db.pool.idleCheckDelay = 0
  + db.mvcc.throwfast = false
  + db.validation = true
- NONTX
  + nonTX.recordUpdate.synch = false
  + nonTX.clusters.sync.immediately = manindex
- TX
  + tx.trackAtomicOperations = false
  + tx.pageCacheSize = 12
- INDEX
  + index.embeddedToSbtreeBonsaiThreshold = 40
  + index.sbtreeBonsaiToEmbeddedThreshold = -1
- HASHTABLE
  + hashTable.slitBucketsBuffer.length = 1500
- INDEX
  + index.auto.synchronousAutoRebuild = true
  + index.auto.lazyUpdates = 10000
  + index.flushAfterCreate = true
  + index.manual.lazyUpdates = 1
  + index.durableInNonTxMode = true
  + index.ignoreNullValuesDefault = false
  + index.txMode = FULL
  + index.cursor.prefetchSize = 500000
- SBTREE
  + sbtree.maxDepth = 64
  + sbtree.maxKeySize = 10240
  + sbtree.maxEmbeddedValueSize = 40960
- SBTREEBONSAI
  + sbtreebonsai.bucketSize = 2
  + sbtreebonsai.linkBagCache.size = 100000
  + sbtreebonsai.linkBagCache.evictionSize = 1000
  + sbtreebonsai.freeSpaceReuseTrigger = 0.5
- RIDBAG
  + ridBag.embeddedDefaultSize = 4
  + ridBag.embeddedToSbtreeBonsaiThreshold = 2147483647
  + ridBag.sbtreeBonsaiToEmbeddedToThreshold = 2147483647
- COLLECTIONS
  + collections.preferSBTreeSet = false
- FILE
  + file.trackFileClose = false
  + file.lock = true
  + file.deleteDelay = 10
  + file.deleteRetry = 50
- SECURITY
  + security.userPasswordSaltIterations = 65536
  + security.userPasswordSaltCacheSize = 500
  + security.userPasswordDefaultAlgorithm = PBKDF2WithHmacSHA256
- NETWORK
  + network.maxConcurrentSessions = 1000
  + network.socketBufferSize = 0
  + network.lockTimeout = 15000
  + network.socketTimeout = 15000
  + network.requestTimeout = 3600000
  + network.retry = 5
  + network.retryDelay = 500
  + network.binary.loadBalancing.enabled = false
  + network.binary.loadBalancing.timeout = 2000
  + network.binary.maxLength = 16384
  + network.binary.readResponse.maxTimes = 20
  + network.binary.debug = false
  + network.http.installDefaultCommands = true
  + network.http.serverInfo = OrientDB Server v.2.2.21 (build 491997a502b8b90210b269824755c56981765034)
  + network.http.maxLength = 1000000
  + network.http.streaming = false
  + network.http.charset = utf-8
  + network.http.jsonResponseError = true
  + network.http.jsonp = false
  + network.http.sessionExpireTimeout = 300
  + network.http.useToken = false
  + network.token.secretKey = 
  + network.token.encryptionAlgorithm = HmacSHA256
  + network.token.expireTimeout = 60
- PROFILER
  + profiler.enabled = false
  + profiler.config = null
  + profiler.autoDump.interval = 0
  + profiler.maxValues = 200
  + profiler.memoryCheckInterval = 120000
- SEQUENCE
  + sequence.maxRetry = 100
  + sequence.retryDelay = 200
- STORAGEPROFILER
  + storageProfiler.intervalBetweenSnapshots = 100
  + storageProfiler.cleanUpInterval = 5000
- LOG
  + log.console.level = info
  + log.file.level = info
- CLASS
  + class.minimumClusters = 0
- LOG
  + log.console.ansi = auto
- CACHE
  + cache.local.impl = com.orientechnologies.orient.core.cache.ORecordCacheWeakRefs
- COMMAND
  + command.timeout = 0
  + command.cache.enabled = false
  + command.cache.evictStrategy = PER_CLUSTER
  + command.cache.minExecutionTime = 10
  + command.cache.maxResultsetSize = 500
- QUERY
  + query.parallelAuto = false
  + query.parallelMinimumRecords = 300000
  + query.parallelResultQueueSize = 20000
  + query.scanPrefetchPages = 20
  + query.scanBatchSize = 1000
  + query.scanThresholdTip = 50000
  + query.limitThresholdTip = 10000
  + query.live.support = true
- STATEMENT
  + statement.cacheSize = 100
- SQL
  + sql.graphConsistencyMode = tx
- CLIENT
  + client.channel.maxPool = 100
  + client.connectionPool.waitTimeout = 5000
  + client.channel.dbReleaseWaitTimeout = 10000
  + client.ssl.enabled = false
  + client.ssl.keyStore = null
  + client.ssl.keyStorePass = null
  + client.ssl.trustStore = null
  + client.ssl.trustStorePass = null
- SERVER
  + server.openAllDatabasesAtStartup = false
  + server.channel.cleanDelay = 5000
  + server.cache.staticFile = false
  + server.log.dumpClientExceptionLevel = FINE
  + server.log.dumpClientExceptionFullStackTrace = false
- DISTRIBUTED
  + distributed.dumpStatsEvery = 0
  + distributed.crudTaskTimeout = 10000
  + distributed.maxStartupDelay = 10000
  + distributed.commandTaskTimeout = 120000
  + distributed.commandQuickTaskTimeout = 5000
  + distributed.commandLongTaskTimeout = 86400000
  + distributed.deployDbTaskTimeout = 1200000
  + distributed.deployChunkTaskTimeout = 15000
  + distributed.deployDbTaskCompression = 7
  + distributed.asynchQueueSize = 0
  + distributed.asynchResponsesTimeout = 15000
  + distributed.purgeResponsesTimerDelay = 15000
  + distributed.conflictResolverRepairerChain = quorum,content,majority,version
  + distributed.conflictResolverRepairerCheckEvery = 5000
  + distributed.conflictResolverRepairerBatch = 50
  + distributed.txAliveTimeout = 30000
  + distributed.requestChannels = 1
  + distributed.responseChannels = 1
  + distributed.heartbeatTimeout = 10000
  + distributed.checkHealthCanOfflineServer = false
  + distributed.checkHealthEvery = 10000
  + distributed.autoRemoveOfflineServers = 0
  + distributed.publishNodeStatusEvery = 10000
  + distributed.localQueueSize = 10000
  + distributed.dbWorkerThreads = 8
  + distributed.queueMaxSize = 10000
  + distributed.backupDirectory = ../backup/databases
  + distributed.backupTryIncrementalFirst = true
  + distributed.concurrentTxMaxAutoRetry = 10
  + distributed.atomicLockTimeout = 50
  + distributed.concurrentTxAutoRetryDelay = 10
- DB
  + db.document.serializer = ORecordSerializerBinary
- CLIENT
  + client.krb5.config = null
  + client.krb5.ccname = null
  + client.krb5.ktname = null
  + client.credentialinterceptor = null
  + client.ci.keyalgorithm = AES
  + client.ci.ciphertransform = AES/CBC/PKCS5Padding
  + client.ci.keystore.file = null
  + client.ci.keystore.password = null
- SECURITY
  + security.createDefaultUsers = true
- SERVER
  + server.security.file = null
- JNA
  + jna.disable.system.library = true
- DISTRIBUTED
  + distributed.queueTimeout = 500000
- DB
  + db.makeFullCheckpointOnIndexChange = true
  + db.makeFullCheckpointOnSchemaChange = true
- CLIENT
  + client.session.tokenBased = true
- OAUTH2
  + oauth2.secretkey = 
- STORAGE
  + storage.cluster.usecrc32 = false
- LAZYSET
  + lazyset.workOnStream = true
- DB
  + db.mvcc = true
  + db.use.distributedVersion = false
- MVRBTREE
  + mvrbtree.timeout = 0
  + mvrbtree.nodePageSize = 256
  + mvrbtree.loadFactor = 0.7
  + mvrbtree.optimizeThreshold = 100000
  + mvrbtree.entryPoints = 64
  + mvrbtree.optimizeEntryPointsFactor = 1.0
  + mvrbtree.entryKeysInMemory = false
  + mvrbtree.entryValuesInMemory = false
  + mvrbtree.ridBinaryThreshold = -1
  + mvrbtree.ridNodePageSize = 64
  + mvrbtree.ridNodeSaveMemory = false
- TX
  + tx.commit.synch = false
  + tx.autoRetry = 1
  + tx.log.fileType = classic
  + tx.log.synch = false
  + tx.useLog = true
- INDEX
  + index.auto.rebuildAfterNotSoftClose = true
- CLIENT
  + client.channel.minPool = 1
- STORAGE
  + storage.keepOpen = true
- CACHE
  + cache.local.enabled = true

OrientDB Memory profiler: HEAP=3.20GB of 3.94GB - DISKCACHE (2 dbs)=3.44GB of 51.17GB - OS=21.08GB of 31.42GB - FS=15.84GB of 98.30GB

THREAD DUMP
"SIGTRAP handler" id=3722975 
   java.lang.Thread.State: RUNNABLE
        at sun.management.ThreadImpl.getThreadInfo1(Native Method)
        at sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:178)
        at com.orientechnologies.common.profiler.OAbstractProfiler.threadDump(OAbstractProfiler.java:432)
        at com.orientechnologies.orient.core.OSignalHandler.handle(OSignalHandler.java:76)
        at sun.misc.Signal$1.run(Signal.java:212)
        at java.lang.Thread.run(Thread.java:748)

"OrientDB HTTP Connection /127.0.0.1:2480<-/127.0.0.1:53258" id=3722974 
   java.lang.Thread.State: RUNNABLE
        at java.lang.Object.notifyAll(Native Method)
        at com.orientechnologies.orient.core.sql.query.OConcurrentResultSet.notifyNewItem(OConcurrentResultSet.java:372)
        at com.orientechnologies.orient.core.sql.query.OConcurrentResultSet.add(OConcurrentResultSet.java:258)
        at com.orientechnologies.orient.core.sql.query.OSQLSynchQuery.result(OSQLSynchQuery.java:68)
        at com.orientechnologies.orient.core.sql.parser.OMatchStatement.addSingleResult(OMatchStatement.java:937)
        at com.orientechnologies.orient.core.sql.parser.OMatchStatement.addResult(OMatchStatement.java:921)
        at com.orientechnologies.orient.core.sql.parser.OMatchStatement.processContext(OMatchStatement.java:573)
        at com.orientechnologies.orient.core.sql.parser.OMatchStatement.processContext(OMatchStatement.java:656)
        at com.orientechnologies.orient.core.sql.parser.OMatchStatement.processContext(OMatchStatement.java:656)
        at com.orientechnologies.orient.core.sql.parser.OMatchStatement.processContextFromCandidates(OMatchStatement.java:546)
        at com.orientechnologies.orient.core.sql.parser.OMatchStatement.calculateMatch(OMatchStatement.java:532)
        at com.orientechnologies.orient.core.sql.parser.OMatchStatement.execute(OMatchStatement.java:269)
        at com.orientechnologies.orient.core.sql.parser.OMatchStatement.execute(OMatchStatement.java:244)
        at com.orientechnologies.orient.core.sql.parser.OMatchStatement.iterator(OMatchStatement.java:1348)
        at com.orientechnologies.orient.core.sql.OCommandExecutorSQLResultsetDelegate.iterator(OCommandExecutorSQLResultsetDelegate.java:44)
        at com.orientechnologies.orient.core.sql.OCommandExecutorSQLResultsetAbstract.assignTarget(OCommandExecutorSQLResultsetAbstract.java:220)
        at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.assignTarget(OCommandExecutorSQLSelect.java:526)
        at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.executeSearch(OCommandExecutorSQLSelect.java:508)
        at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.iterator(OCommandExecutorSQLSelect.java:447)
        at com.orientechnologies.orient.core.sql.OCommandExecutorSQLResultsetDelegate.iterator(OCommandExecutorSQLResultsetDelegate.java:44)
        at com.orientechnologies.orient.core.sql.OCommandExecutorSQLResultsetAbstract.assignTarget(OCommandExecutorSQLResultsetAbstract.java:220)
        at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.assignTarget(OCommandExecutorSQLSelect.java:526)
        at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.executeSearch(OCommandExecutorSQLSelect.java:508)
        at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.execute(OCommandExecutorSQLSelect.java:484)
        at com.orientechnologies.orient.core.sql.OCommandExecutorSQLDelegate.execute(OCommandExecutorSQLDelegate.java:74)
        at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.executeCommand(OAbstractPaginatedStorage.java:3200)
        at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.command(OAbstractPaginatedStorage.java:3138)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedStorage.command(ODistributedStorage.java:280)
        at com.orientechnologies.orient.core.command.OCommandRequestTextAbstract.execute(OCommandRequestTextAbstract.java:69)
        at com.orientechnologies.orient.server.network.protocol.http.command.post.OServerCommandPostBatch.execute(OServerCommandPostBatch.java:171)
        at com.orientechnologies.orient.server.network.protocol.http.ONetworkProtocolHttpAbstract.service(ONetworkProtocolHttpAbstract.java:169)
        at com.orientechnologies.orient.server.network.protocol.http.ONetworkProtocolHttpAbstract.execute(ONetworkProtocolHttpAbstract.java:621)
        at com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:77)

"OrientDB HTTP Connection /127.0.0.1:2480<-/127.0.0.1:53256" id=3722973 
   java.lang.Thread.State: RUNNABLE
        at java.lang.Object.notifyAll(Native Method)
        at com.orientechnologies.orient.core.sql.query.OConcurrentResultSet.notifyNewItem(OConcurrentResultSet.java:372)
        at com.orientechnologies.orient.core.sql.query.OConcurrentResultSet.add(OConcurrentResultSet.java:258)
        at com.orientechnologies.orient.core.sql.query.OSQLSynchQuery.result(OSQLSynchQuery.java:68)
        at com.orientechnologies.orient.core.sql.parser.OMatchStatement.addSingleResult(OMatchStatement.java:937)
        at com.orientechnologies.orient.core.sql.parser.OMatchStatement.addResult(OMatchStatement.java:921)
        at com.orientechnologies.orient.core.sql.parser.OMatchStatement.processContext(OMatchStatement.java:573)
        at com.orientechnologies.orient.core.sql.parser.OMatchStatement.processContext(OMatchStatement.java:656)
        at com.orientechnologies.orient.core.sql.parser.OMatchStatement.processContext(OMatchStatement.java:656)
        at com.orientechnologies.orient.core.sql.parser.OMatchStatement.processContextFromCandidates(OMatchStatement.java:546)
        at com.orientechnologies.orient.core.sql.parser.OMatchStatement.calculateMatch(OMatchStatement.java:532)
        at com.orientechnologies.orient.core.sql.parser.OMatchStatement.execute(OMatchStatement.java:269)
        at com.orientechnologies.orient.core.sql.parser.OMatchStatement.execute(OMatchStatement.java:244)
        at com.orientechnologies.orient.core.sql.parser.OMatchStatement.iterator(OMatchStatement.java:1348)
        at com.orientechnologies.orient.core.sql.OCommandExecutorSQLResultsetDelegate.iterator(OCommandExecutorSQLResultsetDelegate.java:44)
        at com.orientechnologies.orient.core.sql.OCommandExecutorSQLResultsetAbstract.assignTarget(OCommandExecutorSQLResultsetAbstract.java:220)
        at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.assignTarget(OCommandExecutorSQLSelect.java:526)
        at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.executeSearch(OCommandExecutorSQLSelect.java:508)
        at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.iterator(OCommandExecutorSQLSelect.java:447)
        at com.orientechnologies.orient.core.sql.OCommandExecutorSQLResultsetDelegate.iterator(OCommandExecutorSQLResultsetDelegate.java:44)
        at com.orientechnologies.orient.core.sql.OCommandExecutorSQLResultsetAbstract.assignTarget(OCommandExecutorSQLResultsetAbstract.java:220)
        at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.assignTarget(OCommandExecutorSQLSelect.java:526)
        at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.executeSearch(OCommandExecutorSQLSelect.java:508)
        at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.execute(OCommandExecutorSQLSelect.java:484)
        at com.orientechnologies.orient.core.sql.OCommandExecutorSQLDelegate.execute(OCommandExecutorSQLDelegate.java:74)
        at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.executeCommand(OAbstractPaginatedStorage.java:3200)
        at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.command(OAbstractPaginatedStorage.java:3138)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedStorage.command(ODistributedStorage.java:280)
        at com.orientechnologies.orient.core.command.OCommandRequestTextAbstract.execute(OCommandRequestTextAbstract.java:69)
        at com.orientechnologies.orient.server.network.protocol.http.command.post.OServerCommandPostBatch.execute(OServerCommandPostBatch.java:171)
        at com.orientechnologies.orient.server.network.protocol.http.ONetworkProtocolHttpAbstract.service(ONetworkProtocolHttpAbstract.java:169)
        at com.orientechnologies.orient.server.network.protocol.http.ONetworkProtocolHttpAbstract.execute(ONetworkProtocolHttpAbstract.java:621)
        at com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:77)

"OrientDB HTTP Connection /127.0.0.1:2480<-/127.0.0.1:52850" id=3722770 
   java.lang.Thread.State: RUNNABLE
        at com.orientechnologies.orient.core.record.impl.ODocument.field(ODocument.java:823)
        at com.orientechnologies.orient.server.distributed.ODistributedConfiguration.getClusterConfiguration(ODistributedConfiguration.java:754)
        at com.orientechnologies.orient.server.distributed.ODistributedConfiguration.getQuorum(ODistributedConfiguration.java:787)
        at com.orientechnologies.orient.server.distributed.ODistributedConfiguration.getReadQuorum(ODistributedConfiguration.java:708)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedStorage.readRecord(ODistributedStorage.java:757)
        at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx$SimpleRecordReader.readRecord(ODatabaseDocumentTx.java:3361)
        at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.executeReadRecord(ODatabaseDocumentTx.java:2012)
        at com.orientechnologies.orient.core.tx.OTransactionNoTx.loadRecord(OTransactionNoTx.java:106)
        at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.load(ODatabaseDocumentTx.java:1729)
        at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.load(ODatabaseDocumentTx.java:102)
        at com.orientechnologies.orient.core.id.ORecordId.getRecord(ORecordId.java:329)
        at com.tinkerpop.blueprints.impls.orient.OrientVertexIterator.getGraphElementRecord(OrientVertexIterator.java:92)
        at com.orientechnologies.orient.core.iterator.OLazyWrapperIterator.hasNext(OLazyWrapperIterator.java:103)
        at com.orientechnologies.common.collection.OMultiCollectionIterator.hasNextInternal(OMultiCollectionIterator.java:97)
        at com.orientechnologies.common.collection.OMultiCollectionIterator.hasNext(OMultiCollectionIterator.java:78)
        at com.orientechnologies.common.collection.OMultiValue.toSet(OMultiValue.java:824)
        at com.orientechnologies.orient.core.sql.functions.coll.OSQLFunctionIntersect.intersectWith(OSQLFunctionIntersect.java:106)
        at com.orientechnologies.orient.core.sql.functions.coll.OSQLFunctionIntersect.execute(OSQLFunctionIntersect.java:87)
        at com.orientechnologies.orient.core.sql.functions.OSQLFunctionRuntime.execute(OSQLFunctionRuntime.java:128)
        at com.orientechnologies.orient.core.sql.ORuntimeResult.applyRecord(ORuntimeResult.java:141)
        at com.orientechnologies.orient.core.sql.ORuntimeResult.getProjectionResult(ORuntimeResult.java:255)
        at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.addResult(OCommandExecutorSQLSelect.java:703)
        at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.handleResult(OCommandExecutorSQLSelect.java:669)
        at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.executeSearchRecord(OCommandExecutorSQLSelect.java:626)
        at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.serialIterator(OCommandExecutorSQLSelect.java:1615)
        at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.fetchFromTarget(OCommandExecutorSQLSelect.java:1562)
        at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.executeSearch(OCommandExecutorSQLSelect.java:521)
        at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.iterator(OCommandExecutorSQLSelect.java:447)
        at com.orientechnologies.orient.core.sql.OCommandExecutorSQLResultsetDelegate.iterator(OCommandExecutorSQLResultsetDelegate.java:44)
        at com.orientechnologies.orient.core.sql.OCommandExecutorSQLResultsetAbstract.assignTarget(OCommandExecutorSQLResultsetAbstract.java:220)
        at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.assignTarget(OCommandExecutorSQLSelect.java:526)
        at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.executeSearch(OCommandExecutorSQLSelect.java:508)
        at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.execute(OCommandExecutorSQLSelect.java:484)
        at com.orientechnologies.orient.core.sql.OCommandExecutorSQLDelegate.execute(OCommandExecutorSQLDelegate.java:74)
        at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.executeCommand(OAbstractPaginatedStorage.java:3200)
        at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.command(OAbstractPaginatedStorage.java:3138)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedStorage.command(ODistributedStorage.java:280)
        at com.orientechnologies.orient.core.command.OCommandRequestTextAbstract.execute(OCommandRequestTextAbstract.java:69)
        at com.orientechnologies.orient.server.network.protocol.http.command.post.OServerCommandPostBatch.execute(OServerCommandPostBatch.java:171)
        at com.orientechnologies.orient.server.network.protocol.http.ONetworkProtocolHttpAbstract.service(ONetworkProtocolHttpAbstract.java:169)
        at com.orientechnologies.orient.server.network.protocol.http.ONetworkProtocolHttpAbstract.execute(ONetworkProtocolHttpAbstract.java:621)
        at com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:77)

"OrientDB HTTP Connection /127.0.0.1:2480<-/127.0.0.1:49330" id=3721010 
   java.lang.Thread.State: TIMED_WAITING
        at java.lang.Thread.sleep(Native Method)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedTransactionManager.acquireMultipleRecordLocks(ODistributedTransactionManager.java:546)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedTransactionManager.acquireMultipleRecordLocks(ODistributedTransactionManager.java:512)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedTransactionManager.commit(ODistributedTransactionManager.java:108)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedStorage.commit(ODistributedStorage.java:1419)
        at com.orientechnologies.orient.core.tx.OTransactionOptimistic.doCommit(OTransactionOptimistic.java:541)
        at com.orientechnologies.orient.core.tx.OTransactionOptimistic.commit(OTransactionOptimistic.java:99)
        at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:2886)
        at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:2855)
        at com.tinkerpop.blueprints.impls.orient.OrientTransactionalGraph.commit(OrientTransactionalGraph.java:182)
        at com.orientechnologies.orient.graph.sql.OGraphCommandExecutorSQLFactory.runInConfiguredTxMode(OGraphCommandExecutorSQLFactory.java:265)
        at com.orientechnologies.orient.graph.sql.OCommandExecutorSQLCreateEdge.execute(OCommandExecutorSQLCreateEdge.java:143)
        at com.orientechnologies.orient.core.sql.OCommandExecutorSQLDelegate.execute(OCommandExecutorSQLDelegate.java:74)
        at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.executeCommand(OAbstractPaginatedStorage.java:3200)
        at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.command(OAbstractPaginatedStorage.java:3138)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedStorage.command(ODistributedStorage.java:260)
        at com.orientechnologies.orient.core.command.OCommandRequestTextAbstract.execute(OCommandRequestTextAbstract.java:69)
        at com.orientechnologies.orient.server.network.protocol.http.command.post.OServerCommandPostBatch.execute(OServerCommandPostBatch.java:171)
        at com.orientechnologies.orient.server.network.protocol.http.ONetworkProtocolHttpAbstract.service(ONetworkProtocolHttpAbstract.java:169)
        at com.orientechnologies.orient.server.network.protocol.http.ONetworkProtocolHttpAbstract.execute(ONetworkProtocolHttpAbstract.java:621)
        at com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:77)

"cached12" id=3712842 
   java.lang.Thread.State: WAITING
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
        at com.hazelcast.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76)
        at com.hazelcast.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:92)

"cached1" id=3205580 
   java.lang.Thread.State: WAITING
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
        at com.hazelcast.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76)
        at com.hazelcast.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:92)

"OrientDB (/172.31.4.100:2424) <- BinaryClient (/172.31.17.48:51610)" id=3056174 
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
        at java.io.DataInputStream.readByte(DataInputStream.java:265)
        at com.orientechnologies.orient.enterprise.channel.binary.OChannelBinary.readByte(OChannelBinary.java:68)
        at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.execute(ONetworkProtocolBinary.java:189)
        at com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:77)

"cached9" id=3036212 
   java.lang.Thread.State: WAITING
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
        at com.hazelcast.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76)
        at com.hazelcast.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:92)

"OrientDB HTTP Connection /172.31.4.100:2480<-/182.71.117.210:48594" id=2876562 
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
        at com.orientechnologies.orient.enterprise.channel.text.OChannelText.read(OChannelText.java:61)
        at com.orientechnologies.orient.server.network.protocol.http.ONetworkProtocolHttpAbstract.execute(ONetworkProtocolHttpAbstract.java:565)
        at com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:77)

"OrientDB Distributed asynch ops node=node1 db=db_development" id=2284840 
   java.lang.Thread.State: WAITING
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedStorage$1.run(ODistributedStorage.java:136)

"OrientDB Write Cache Flush Task (db_development)" id=2284837 
   java.lang.Thread.State: TIMED_WAITING
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)

"OrientDB WAL Flush Task (db_development)" id=2284836 
   java.lang.Thread.State: TIMED_WAITING
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)

"WAL Closer Task (db_development)" id=2284835 
   java.lang.Thread.State: TIMED_WAITING
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)

"OrientDB (/172.31.4.100:2424) <- BinaryClient (/172.31.17.48:56562)" id=2284829 
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
        at java.io.DataInputStream.readByte(DataInputStream.java:265)
        at com.orientechnologies.orient.enterprise.channel.binary.OChannelBinary.readByte(OChannelBinary.java:68)
        at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.execute(ONetworkProtocolBinary.java:189)
        at com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:77)

"hz._hzInstance_1_root.IO.thread-Acceptor" id=2284812 
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
        at com.hazelcast.nio.tcp.SocketAcceptorThread.acceptLoop(SocketAcceptorThread.java:92)
        at com.hazelcast.nio.tcp.SocketAcceptorThread.run(SocketAcceptorThread.java:78)

"hz._hzInstance_1_root.IOBalancerThread" id=2284811 
   java.lang.Thread.State: TIMED_WAITING
        at java.lang.Thread.sleep(Native Method)
        at java.lang.Thread.sleep(Thread.java:340)
        at java.util.concurrent.TimeUnit.sleep(TimeUnit.java:386)
        at com.hazelcast.nio.tcp.nonblocking.iobalancer.IOBalancerThread.run(IOBalancerThread.java:53)

"hz._hzInstance_1_root.IO.thread-out-2" id=2284810 
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at com.hazelcast.nio.tcp.nonblocking.NonBlockingIOThread.selectLoop(NonBlockingIOThread.java:232)
        at com.hazelcast.nio.tcp.nonblocking.NonBlockingIOThread.run(NonBlockingIOThread.java:187)

"hz._hzInstance_1_root.IO.thread-out-1" id=2284809 
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at com.hazelcast.nio.tcp.nonblocking.NonBlockingIOThread.selectLoop(NonBlockingIOThread.java:232)
        at com.hazelcast.nio.tcp.nonblocking.NonBlockingIOThread.run(NonBlockingIOThread.java:187)

"hz._hzInstance_1_root.IO.thread-out-0" id=2284808 
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at com.hazelcast.nio.tcp.nonblocking.NonBlockingIOThread.selectLoop(NonBlockingIOThread.java:232)
        at com.hazelcast.nio.tcp.nonblocking.NonBlockingIOThread.run(NonBlockingIOThread.java:187)

"hz._hzInstance_1_root.IO.thread-in-2" id=2284807 
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at com.hazelcast.nio.tcp.nonblocking.NonBlockingIOThread.selectLoop(NonBlockingIOThread.java:232)
        at com.hazelcast.nio.tcp.nonblocking.NonBlockingIOThread.run(NonBlockingIOThread.java:187)

"hz._hzInstance_1_root.IO.thread-in-1" id=2284806 
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at com.hazelcast.nio.tcp.nonblocking.NonBlockingIOThread.selectLoop(NonBlockingIOThread.java:232)
        at com.hazelcast.nio.tcp.nonblocking.NonBlockingIOThread.run(NonBlockingIOThread.java:187)

"hz._hzInstance_1_root.IO.thread-in-0" id=2284805 
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at com.hazelcast.nio.tcp.nonblocking.NonBlockingIOThread.selectLoop(NonBlockingIOThread.java:232)
        at com.hazelcast.nio.tcp.nonblocking.NonBlockingIOThread.run(NonBlockingIOThread.java:187)

"OrientDB DistributedWorker node=node1 db=db_development id=7" id=73 
   java.lang.Thread.State: TIMED_WAITING
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.nextMessage(ODistributedWorker.java:252)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.readRequest(ODistributedWorker.java:229)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.run(ODistributedWorker.java:93)

"OrientDB DistributedWorker node=node1 db=db_development id=6" id=72 
   java.lang.Thread.State: TIMED_WAITING
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.nextMessage(ODistributedWorker.java:252)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.readRequest(ODistributedWorker.java:229)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.run(ODistributedWorker.java:93)

"OrientDB DistributedWorker node=node1 db=db_development id=5" id=71 
   java.lang.Thread.State: TIMED_WAITING
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.nextMessage(ODistributedWorker.java:252)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.readRequest(ODistributedWorker.java:229)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.run(ODistributedWorker.java:93)

"OrientDB DistributedWorker node=node1 db=db_development id=4" id=70 
   java.lang.Thread.State: TIMED_WAITING
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.nextMessage(ODistributedWorker.java:252)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.readRequest(ODistributedWorker.java:229)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.run(ODistributedWorker.java:93)

"OrientDB DistributedWorker node=node1 db=db_development id=3" id=69 
   java.lang.Thread.State: TIMED_WAITING
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.nextMessage(ODistributedWorker.java:252)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.readRequest(ODistributedWorker.java:229)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.run(ODistributedWorker.java:93)

"OrientDB DistributedWorker node=node1 db=db_development id=2" id=68 
   java.lang.Thread.State: TIMED_WAITING
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.nextMessage(ODistributedWorker.java:252)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.readRequest(ODistributedWorker.java:229)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.run(ODistributedWorker.java:93)

"OrientDB DistributedWorker node=node1 db=db_development id=1" id=67 
   java.lang.Thread.State: TIMED_WAITING
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.nextMessage(ODistributedWorker.java:252)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.readRequest(ODistributedWorker.java:229)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.run(ODistributedWorker.java:93)

"OrientDB DistributedWorker node=node1 db=db_development id=0" id=66 
   java.lang.Thread.State: TIMED_WAITING
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.nextMessage(ODistributedWorker.java:252)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.readRequest(ODistributedWorker.java:229)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.run(ODistributedWorker.java:93)

"OrientDB DistributedWorker node=node1 db=db_development id=-4" id=65 
   java.lang.Thread.State: TIMED_WAITING
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.nextMessage(ODistributedWorker.java:252)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.readRequest(ODistributedWorker.java:229)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.run(ODistributedWorker.java:93)

"OrientDB DistributedWorker node=node1 db=db_development id=-3" id=64 
   java.lang.Thread.State: TIMED_WAITING
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.nextMessage(ODistributedWorker.java:252)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.readRequest(ODistributedWorker.java:229)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.run(ODistributedWorker.java:93)

"OrientDB DistributedWorker node=node1 db=OSystem id=7" id=63 
   java.lang.Thread.State: TIMED_WAITING
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.nextMessage(ODistributedWorker.java:252)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.readRequest(ODistributedWorker.java:229)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.run(ODistributedWorker.java:93)

"OrientDB DistributedWorker node=node1 db=OSystem id=6" id=62 
   java.lang.Thread.State: TIMED_WAITING
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.nextMessage(ODistributedWorker.java:252)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.readRequest(ODistributedWorker.java:229)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.run(ODistributedWorker.java:93)

"OrientDB DistributedWorker node=node1 db=OSystem id=5" id=61 
   java.lang.Thread.State: TIMED_WAITING
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.nextMessage(ODistributedWorker.java:252)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.readRequest(ODistributedWorker.java:229)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.run(ODistributedWorker.java:93)

"OrientDB DistributedWorker node=node1 db=OSystem id=4" id=60 
   java.lang.Thread.State: TIMED_WAITING
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.nextMessage(ODistributedWorker.java:252)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.readRequest(ODistributedWorker.java:229)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.run(ODistributedWorker.java:93)

"OrientDB DistributedWorker node=node1 db=OSystem id=3" id=59 
   java.lang.Thread.State: TIMED_WAITING
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.nextMessage(ODistributedWorker.java:252)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.readRequest(ODistributedWorker.java:229)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.run(ODistributedWorker.java:93)

"OrientDB DistributedWorker node=node1 db=OSystem id=2" id=58 
   java.lang.Thread.State: TIMED_WAITING
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.nextMessage(ODistributedWorker.java:252)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.readRequest(ODistributedWorker.java:229)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.run(ODistributedWorker.java:93)

"OrientDB DistributedWorker node=node1 db=OSystem id=1" id=57 
   java.lang.Thread.State: TIMED_WAITING
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.nextMessage(ODistributedWorker.java:252)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.readRequest(ODistributedWorker.java:229)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.run(ODistributedWorker.java:93)

"OrientDB DistributedWorker node=node1 db=OSystem id=0" id=56 
   java.lang.Thread.State: TIMED_WAITING
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.nextMessage(ODistributedWorker.java:252)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.readRequest(ODistributedWorker.java:229)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.run(ODistributedWorker.java:93)

"OrientDB DistributedWorker node=node1 db=OSystem id=-4" id=55 
   java.lang.Thread.State: TIMED_WAITING
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.nextMessage(ODistributedWorker.java:252)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.readRequest(ODistributedWorker.java:229)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.run(ODistributedWorker.java:93)

"OrientDB DistributedWorker node=node1 db=OSystem id=-3" id=54 
   java.lang.Thread.State: TIMED_WAITING
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.nextMessage(ODistributedWorker.java:252)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.readRequest(ODistributedWorker.java:229)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.run(ODistributedWorker.java:93)

"hz._hzInstance_1_root.HealthMonitor" id=53 
   java.lang.Thread.State: TIMED_WAITING
        at java.lang.Thread.sleep(Native Method)
        at java.lang.Thread.sleep(Thread.java:340)
        at java.util.concurrent.TimeUnit.sleep(TimeUnit.java:386)
        at com.hazelcast.internal.monitors.HealthMonitor$HealthMonitorThread.run(HealthMonitor.java:135)

"hz._hzInstance_1_root.generic-operation.thread-0" id=30 
   java.lang.Thread.State: WAITING
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at com.hazelcast.spi.impl.operationexecutor.classic.DefaultScheduleQueue.take(DefaultScheduleQueue.java:84)
        at com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.doRun(OperationThread.java:112)
        at com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.run(OperationThread.java:99)

"hz._hzInstance_1_root.partition-operation.thread-0" id=29 
   java.lang.Thread.State: WAITING
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at com.hazelcast.spi.impl.operationexecutor.classic.DefaultScheduleQueue.take(DefaultScheduleQueue.java:84)
        at com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.doRun(OperationThread.java:112)
        at com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.run(OperationThread.java:99)

"hz._hzInstance_1_root.migration" id=34 
   java.lang.Thread.State: TIMED_WAITING
        at java.lang.Thread.sleep(Native Method)
        at com.hazelcast.partition.impl.InternalPartitionServiceImpl$MigrationThread.doRun(InternalPartitionServiceImpl.java:2137)
        at com.hazelcast.partition.impl.InternalPartitionServiceImpl$MigrationThread.run(InternalPartitionServiceImpl.java:2101)

"hz._hzInstance_1_root.wait-notify" id=33 
   java.lang.Thread.State: TIMED_WAITING
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.DelayQueue.poll(DelayQueue.java:259)
        at com.hazelcast.spi.impl.waitnotifyservice.impl.WaitNotifyServiceImpl$ExpirationTask.doRun(WaitNotifyServiceImpl.java:311)
        at com.hazelcast.spi.impl.waitnotifyservice.impl.WaitNotifyServiceImpl$ExpirationTask.run(WaitNotifyServiceImpl.java:296)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
        at com.hazelcast.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76)
        at com.hazelcast.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:92)

"hz._hzInstance_1_root.event-1" id=32 
   java.lang.Thread.State: WAITING
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at com.hazelcast.util.executor.StripedExecutor$Worker.run(StripedExecutor.java:170)

"hz._hzInstance_1_root.SlowOperationDetectorThread" id=31 
   java.lang.Thread.State: TIMED_WAITING
        at java.lang.Thread.sleep(Native Method)
        at java.lang.Thread.sleep(Thread.java:340)
        at java.util.concurrent.TimeUnit.sleep(TimeUnit.java:386)
        at com.hazelcast.spi.impl.operationexecutor.slowoperationdetector.SlowOperationDetector$DetectorThread.sleepInterval(SlowOperationDetector.java:287)
        at com.hazelcast.spi.impl.operationexecutor.slowoperationdetector.SlowOperationDetector$DetectorThread.run(SlowOperationDetector.java:151)

"hz._hzInstance_1_root.response" id=28 
   java.lang.Thread.State: WAITING
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at com.hazelcast.spi.impl.operationservice.impl.AsyncResponsePacketHandler$ResponseThread.doRun(AsyncResponsePacketHandler.java:110)
        at com.hazelcast.spi.impl.operationservice.impl.AsyncResponsePacketHandler$ResponseThread.run(AsyncResponsePacketHandler.java:99)

"hz._hzInstance_1_root.InvocationMonitorThread" id=27 
   java.lang.Thread.State: TIMED_WAITING
        at java.lang.Thread.sleep(Native Method)
        at com.hazelcast.spi.impl.operationservice.impl.InvocationMonitor$MonitorThread.sleep(InvocationMonitor.java:145)
        at com.hazelcast.spi.impl.operationservice.impl.InvocationMonitor$MonitorThread.run(InvocationMonitor.java:134)

"scheduled" id=26 
   java.lang.Thread.State: TIMED_WAITING
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
        at com.hazelcast.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76)
        at com.hazelcast.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:92)
"pool-2-thread-2" id=25 
   java.lang.Thread.State: WAITING
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)

"pool-2-thread-1" id=24 
   java.lang.Thread.State: TIMED_WAITING
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)

"OrientDB Write Cache Flush Task (OSystem)" id=21 
   java.lang.Thread.State: TIMED_WAITING
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)

"OrientDB WAL Flush Task (OSystem)" id=20 
   java.lang.Thread.State: TIMED_WAITING
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)

"WAL Closer Task (OSystem)" id=19 
   java.lang.Thread.State: TIMED_WAITING
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)

"OrientDB ONetworkProtocolHttpDb listen at 0.0.0.0:2480-2490" id=17 
   java.lang.Thread.State: RUNNABLE
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
        at java.net.ServerSocket.implAccept(ServerSocket.java:545)
        at java.net.ServerSocket.accept(ServerSocket.java:513)
        at com.orientechnologies.orient.server.network.OServerNetworkListener.run(OServerNetworkListener.java:189)

"OrientDB ONetworkProtocolBinary listen at 0.0.0.0:2424-2430" id=15 
   java.lang.Thread.State: RUNNABLE
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
        at java.net.ServerSocket.implAccept(ServerSocket.java:545)
        at java.net.ServerSocket.accept(ServerSocket.java:513)
        at com.orientechnologies.orient.server.network.OServerNetworkListener.run(OServerNetworkListener.java:189)

"Timer-0" id=12 
   java.lang.Thread.State: TIMED_WAITING
        at java.lang.Object.wait(Native Method)
        at java.util.TimerThread.mainLoop(Timer.java:552)
        at java.util.TimerThread.run(Timer.java:505)
"Thread-0" id=10 
   java.lang.Thread.State: WAITING
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
        at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
        at com.orientechnologies.orient.server.OServer.waitForShutdown(OServer.java:528)
        at com.orientechnologies.orient.server.OServerMain$1.run(OServerMain.java:47)

"Signal Dispatcher" id=4 
   java.lang.Thread.State: RUNNABLE

"Finalizer" id=3 
   java.lang.Thread.State: WAITING
        at java.lang.Object.wait(Native Method)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)

"Reference Handler" id=2 
   java.lang.Thread.State: WAITING
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)

"main" id=1 
   java.lang.Thread.State: WAITING
        at java.lang.Object.wait(Native Method)
        at java.lang.Thread.join(Thread.java:1252)
        at java.lang.Thread.join(Thread.java:1326)
        at com.orientechnologies.orient.server.OServerMain.main(OServerMain.java:57)



REPLICATION LATENCY AVERAGE (in milliseconds)
+-------+------+-----+
|Servers|node1*|node2|
+-------+------+-----+
|node1* |      | 0.73|
|node2  |  0.67|     |
+-------+------+-----+

REPLICATION MESSAGE COUNTERS (servers: source on the row and destination on the column)
+-------+---------+----------+----------+
|Servers|   node1*|     node2|     TOTAL|
+-------+---------+----------+----------+
|node1* |         |12,162,352|12,162,352|
|node2  |1,896,763|          | 1,896,763|
+-------+---------+----------+----------+
|TOTAL  |1,896,763|12,162,352|14,059,115|
+-------+---------+----------+----------+

REPLICATION MESSAGE CURRENT NODE STATS
+------+-------+-------+--------+---------+----------+----------+-----------+------------+-------------+-------------+-------------+-------------+--------------+
|    tx|Servers| gossip|exc_lock|deploy_db|upd_db_cfg|     TOTAL|command_sql|tx-completed|record_update|copy_db_chunk|record_create|repair_record|deploy_delta_d|
+------+-------+-------+--------+---------+----------+----------+-----------+------------+-------------+-------------+-------------+-------------+--------------+
|34,744|node1* | 53,792|       8|        2|         4|12,162,559|          1|      38,676|   12,026,216|          234|        4,948|        3,932|             2|
|    18|node2  | 53,896|       2|         |         2| 3,445,070|          1|          21|    3,391,126|             |             |            3|             1|
+------+-------+-------+--------+---------+----------+----------+-----------+------------+-------------+-------------+-------------+-------------+--------------+
|34,762|TOTAL  |107,688|      10|        2|         6|15,607,629|          2|      38,697|   15,417,342|          234|        4,948|        3,935|             3|
+------+-------+-------+--------+---------+----------+----------+-----------+------------+-------------+-------------+-------------+-------------+--------------+


DATABASE 'db_development' ON SERVER 'node1'
- HA RESOURCE LOCKS FOR SERVER 'node1'
- HA RECORD LOCKS FOR DATABASE 'db_development'
+----------+------+-----------------------+----------+-----------+
|rid       |server|acquiredOn             |reqId     |threadCount|
+----------+------+-----------------------+----------+-----------+
|#12:5470  |node1 |2017-07-10 04:48:06:735|0.17009966|1          |
|#12:5544  |node1 |2017-07-10 04:33:02:517|0.14821725|1          |
|#12:5760  |node1 |2017-07-10 04:27:25:125|0.13881368|1          |
|#12:7498  |node1 |2017-07-09 17:00:58:168|0.12585373|1          |
|#12:8981  |node1 |2017-07-07 17:01:40:835|0.10516242|1          |
|#12:271967|node1 |2017-07-10 12:47:07:750|0.24484296|1          |
|#12:443953|node1 |2017-07-10 12:07:22:760|0.24458014|1          |
|#13:111   |node1 |2017-07-10 12:13:29:786|0.24468003|1          |
+----------+------+-----------------------+----------+-----------+

- MESSAGES IN QUEUES:
 - QUEUE LOCK EXECUTING: null
 - QUEUE UNLOCK EXECUTING: null
 - QUEUE 0 EXECUTING: null
 - QUEUE 1 EXECUTING: null
 - QUEUE 2 EXECUTING: null
 - QUEUE 3 EXECUTING: null
 - QUEUE 4 EXECUTING: null
 - QUEUE 5 EXECUTING: null
 - QUEUE 6 EXECUTING: null
 - QUEUE 7 EXECUTING: null

@lvca Can you please look at the log and tell us where are we going wrong.

We added the following code as you suggested:

for( int retry = 0; retry < 10; ++retry ){
 try {
  // EXECUTE YOUR COMMAND
  break;

 } catch( ONeedRetryException e ){
   // CONTINUE
 }
}

But the retry continues for indefinite time and the record is not unlocked. So this does not solve our issue.

roots-ai · 2017-07-11T11:09:39Z

Hi @lvca ! We've shared the output above, please help.

nishantkumar1292 · 2017-07-12T06:27:40Z

We also have another major issue here!!
The contents of our database get erased without our knowledge and this is happening frequently. This generally happens when there is the database is not shut down properly or heavy query is run on the database. Also sometimes, the query gets interrupted due to some reason and when we restart the server, the database folder is empty.
@lvca ..What is happening here?

lvca · 2017-07-12T07:08:53Z

@nishantkumar1292 what do you mean for erased? What content? Records, Clusters or the entire database? I'm sure in the log you can find the reason why the folder is empty, no space left on device or lack of permissions to the directory? Please send logs, otherwise it's impossible to receive help.

lvca · 2017-07-12T07:15:37Z

About this issue, I see there is a problem here:

+----------+------+-----------------------+----------+-----------+
|rid       |server|acquiredOn             |reqId     |threadCount|
+----------+------+-----------------------+----------+-----------+
|#12:5470  |node1 |2017-07-10 04:48:06:735|0.17009966|1          |
|#12:5544  |node1 |2017-07-10 04:33:02:517|0.14821725|1          |
|#12:5760  |node1 |2017-07-10 04:27:25:125|0.13881368|1          |
|#12:7498  |node1 |2017-07-09 17:00:58:168|0.12585373|1          |
|#12:8981  |node1 |2017-07-07 17:01:40:835|0.10516242|1          |
|#12:271967|node1 |2017-07-10 12:47:07:750|0.24484296|1          |
|#12:443953|node1 |2017-07-10 12:07:22:760|0.24458014|1          |
|#13:111   |node1 |2017-07-10 12:13:29:786|0.24468003|1          |
+----------+------+-----------------------+----------+-----------+

The locks are kept for way too much time causing this problem. It looks like the background thread that checks the timeout is not working. I'm investigating it.

lvca · 2017-07-12T07:18:16Z

Question: is it possible you're running some commands non in transaction?

nishantkumar1292 · 2017-07-12T09:53:33Z

@lvca Answer to your question: Earlier we were running the queries with transaction=true but later to solve ODistributedRecordLockedException we made transaction=false. So currently we are running commands not in transaction.

About emptying of database: The database folder gets erased - all the contents of the folder. The folder is there but the contents are erased.

2017-07-11 13:06:59:773 INFO  Loading configuration from: /home/ubuntu/orientdb-community-2.2.21/config/orientdb-server-config.xml... [OServerConfigurationLoaderXml]
2017-07-11 13:06:59:950 INFO  OrientDB Server v2.2.21 (build 491997a502b8b90210b269824755c56981765034) is starting up... [OServer]
2017-07-11 13:06:59:955 INFO  Databases directory: /home/ubuntu/data/databases [OServer]
2017-07-11 13:06:59:993 INFO  OrientDB auto-config DISKCACHE=26,200MB (heap=3,925MB direct=524,288MB os=32,174MB) [OMemoryAndLocalPaginatedEnginesInitializer]
2017-07-11 13:07:00:123 INFO  Listening binary connections on 0.0.0.0:2424 (protocol v.36, socket=default) [OServerNetworkListener]
2017-07-11 13:07:00:126 INFO  Listening http connections on 0.0.0.0:2480 (protocol v.10, socket=default) [OServerNetworkListener]
2017-07-11 13:07:00:132 INFO  Installing dynamic plugin 'orientdb-studio-2.2.21.zip'... [OServerPluginManager]
2017-07-11 13:07:00:135 INFO  ODefaultPasswordAuthenticator is active [ODefaultPasswordAuthenticator]
2017-07-11 13:07:00:136 INFO  OServerConfigAuthenticator is active [OServerConfigAuthenticator]
2017-07-11 13:07:00:136 INFO  OSystemUserAuthenticator is active [OSystemUserAuthenticator]
2017-07-11 13:07:00:151 INFO  Installed GREMLIN language v.2.6.0 - graph.pool.max=50 [OGraphServerHandler]$ANSI{green {db=OSystem}} Exception `1FC53E33` in storage `OSystem`
com.orientechnologies.orient.core.exception.OStorageException: Cannot open local storage '/home/ubuntu/data/databases/OSystem' with mode=rw
	DB name="OSystem"
	at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.open(OAbstractPaginatedStorage.java:299)
	at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.open(ODatabaseDocumentTx.java:258)
	at com.orientechnologies.orient.server.OServer.openDatabaseBypassingSecurity(OServer.java:980)
	at com.orientechnologies.orient.server.OServer.openDatabase(OServer.java:947)
	at com.orientechnologies.orient.server.OServer.openDatabase(OServer.java:932)
	at com.orientechnologies.orient.server.OSystemDatabase.openSystemDatabase(OSystemDatabase.java:92)
	at com.orientechnologies.orient.server.OSystemDatabase.execute(OSystemDatabase.java:99)
	at com.orientechnologies.orient.server.security.ODefaultServerSecurity.getSystemUser(ODefaultServerSecurity.java:284)
	at com.orientechnologies.orient.server.security.authenticator.OSystemUserAuthenticator.getUser(OSystemUserAuthenticator.java:126)
	at com.orientechnologies.orient.server.security.ODefaultServerSecurity.getUser(ODefaultServerSecurity.java:411)
	at com.orientechnologies.orient.server.OServer.getUser(OServer.java:773)
	at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.config(ODistributedAbstractPlugin.java:172)
	at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.config(OHazelcastPlugin.java:107)
	at com.orientechnologies.orient.server.OServer.registerPlugins(OServer.java:1233)
	at com.orientechnologies.orient.server.OServer.activate(OServer.java:397)
	at com.orientechnologies.orient.server.OServerMain$1.run(OServerMain.java:46)
Caused by: com.orientechnologies.orient.core.exception.OStorageException: Can not open storage it is acquired by other process
	DB name="OSystem"
	at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedStorageDirtyFlag.lockFile(OPaginatedStorageDirtyFlag.java:96)
	at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedStorageDirtyFlag.open(OPaginatedStorageDirtyFlag.java:129)
	at com.orientechnologies.orient.core.storage.impl.local.paginated.OLocalPaginatedStorage.preOpenSteps(OLocalPaginatedStorage.java:395)
	at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.open(OAbstractPaginatedStorage.java:250)
	... 15 more

2017-07-11 13:07:00:207 SEVER getUser() Exception: Cannot open local storage '/home/ubuntu/data/databases/OSystem' with mode=rw
	DB name="OSystem" [OSystemUserAuthenticator]
2017-07-11 13:07:00:247 INFO  Configuring Hazelcast from '/home/ubuntu/orientdb-community-2.2.21/config/hazelcast.xml'. [FileSystemXmlConfig]
2017-07-11 13:07:00:283 WARNI Name of the hazelcast schema location incorrect using default [AbstractXmlConfigHelper]
2017-07-11 13:07:00:502 INFO  [LOCAL] [root] [3.6.5] Interfaces is disabled, trying to pick one address from TCP-IP config addresses: [10.2.2.2, 10.1.1.1] [DefaultAddressPicker]
2017-07-11 13:07:00:502 INFO  [LOCAL] [root] [3.6.5] Prefer IPv4 stack is true. [DefaultAddressPicker]
2017-07-11 13:07:00:508 INFO  [LOCAL] [root] [3.6.5] Picked Address[10.1.1.1]:2481, using socket ServerSocket[addr=/0:0:0:0:0:0:0:0,localport=2481], bind any local is true [DefaultAddressPicker]
2017-07-11 13:07:00:518 INFO  [10.1.1.1]:2481 [root] [3.6.5] Hazelcast 3.6.5 (20160823 - e4af3d9) starting at Address[10.1.1.1]:2481 [system]
2017-07-11 13:07:00:518 INFO  [10.1.1.1]:2481 [root] [3.6.5] Copyright (c) 2008-2016, Hazelcast, Inc. All Rights Reserved. [system]
2017-07-11 13:07:00:518 INFO  [10.1.1.1]:2481 [root] [3.6.5] Configured Hazelcast Serialization version : 1 [system]
2017-07-11 13:07:00:660 INFO  [10.1.1.1]:2481 [root] [3.6.5] Backpressure is disabled [OperationService]
2017-07-11 13:07:00:678 INFO  [10.1.1.1]:2481 [root] [3.6.5] Starting with 1 generic operation threads and 1 partition operation threads. [ClassicOperationExecutor]
2017-07-11 13:07:00:979 INFO  [10.1.1.1]:2481 [root] [3.6.5] Creating TcpIpJoiner [Node]
2017-07-11 13:07:00:980 INFO  [10.1.1.1]:2481 [root] [3.6.5] Address[10.1.1.1]:2481 is STARTING [LifecycleService]
2017-07-11 13:07:01:048 INFO  [10.1.1.1]:2481 [root] [3.6.5] TcpIpConnectionManager configured with Non Blocking IO-threading model: 3 input threads and 3 output threads [NonBlockingIOThreadingModel]
2017-07-11 13:07:01:066 INFO  [10.1.1.1]:2481 [root] [3.6.5] Connecting to /10.2.2.2:2482, timeout: 0, bind-any: true [InitConnectionTask]
2017-07-11 13:07:01:067 INFO  [10.1.1.1]:2481 [root] [3.6.5] Connecting to /10.2.2.2:2481, timeout: 0, bind-any: true [InitConnectionTask]
2017-07-11 13:07:01:067 INFO  [10.1.1.1]:2481 [root] [3.6.5] Connecting to /10.1.1.1:2480, timeout: 0, bind-any: true [InitConnectionTask]
2017-07-11 13:07:01:067 INFO  [10.1.1.1]:2481 [root] [3.6.5] Connecting to /10.2.2.2:2480, timeout: 0, bind-any: true [InitConnectionTask]
2017-07-11 13:07:01:068 INFO  [10.1.1.1]:2481 [root] [3.6.5] Connecting to /10.1.1.1:2482, timeout: 0, bind-any: true [InitConnectionTask]
2017-07-11 13:07:01:068 INFO  [10.1.1.1]:2481 [root] [3.6.5] Could not connect to: /10.2.2.2:2480. Reason: SocketException[Connection refused to address /10.2.2.2:2480] [InitConnectionTask]
2017-07-11 13:07:01:068 INFO  [10.1.1.1]:2481 [root] [3.6.5] Could not connect to: /10.1.1.1:2482. Reason: SocketException[Connection refused to address /10.1.1.1:2482] [InitConnectionTask]
2017-07-11 13:07:01:069 INFO  [10.1.1.1]:2481 [root] [3.6.5] Address[10.1.1.1]:2482 is added to the blacklist. [TcpIpJoiner]
2017-07-11 13:07:01:069 INFO  [10.1.1.1]:2481 [root] [3.6.5] Could not connect to: /10.2.2.2:2482. Reason: SocketException[Connection refused to address /10.2.2.2:2482] [InitConnectionTask]
2017-07-11 13:07:01:069 INFO  [10.1.1.1]:2481 [root] [3.6.5] Could not connect to: /10.2.2.2:2481. Reason: SocketException[Connection refused to address /10.2.2.2:2481] [InitConnectionTask]
2017-07-11 13:07:01:069 INFO  [10.1.1.1]:2481 [root] [3.6.5] Address[10.2.2.2]:2481 is added to the blacklist. [TcpIpJoiner]
2017-07-11 13:07:01:070 INFO  [10.1.1.1]:2481 [root] [3.6.5] Address[10.2.2.2]:2482 is added to the blacklist. [TcpIpJoiner]
2017-07-11 13:07:01:070 INFO  [10.1.1.1]:2481 [root] [3.6.5] Address[10.2.2.2]:2480 is added to the blacklist. [TcpIpJoiner]
2017-07-11 13:07:01:074 INFO  [10.1.1.1]:2481 [root] [3.6.5] Established socket connection between /10.1.1.1:57901 and /10.1.1.1:2480 [TcpIpConnectionManager]
2017-07-11 13:07:06:062 INFO  [10.1.1.1]:2481 [root] [3.6.5] 


Members [1] {
	Member [10.1.1.1]:2481 this
}
 [TcpIpJoiner]
2017-07-11 13:07:06:062 WARNI [10.1.1.1]:2481 [root] [3.6.5] Config seed port is 2480 and cluster size is 1. Some of the ports seem occupied! [Node]
2017-07-11 13:07:06:085 INFO  [10.1.1.1]:2481 [root] [3.6.5] Address[10.1.1.1]:2481 is STARTED [LifecycleService]
2017-07-11 13:07:06:086 INFO  Starting distributed server 'node1' (hzID=6898f945-c104-4980-9b05-3f516b7c627c)... [OHazelcastPlugin]
2017-07-11 13:07:06:089 INFO  Distributed cluster time=Tue Jul 11 13:07:06 UTC 2017 (delta from local node=0)... [OHazelcastPlugin]
2017-07-11 13:07:06:109 INFO  [10.1.1.1]:2481 [root] [3.6.5] Initializing cluster partition table arrangement... [InternalPartitionService]
2017-07-11 13:07:06:170 INFO  [node1] Registered local server with nodeId=0 [OHazelcastPlugin]
2017-07-11 13:07:06:187 INFO  Distributed Lock Manager server is 'node1' [OHazelcastPlugin]
2017-07-11 13:07:06:194 INFO  [node1] Loaded configuration for database 'OSystem' from disk: /home/ubuntu/orientdb-community-2.2.21/config/default-distributed-db-config.json [ODistributedStorage]
2017-07-11 13:07:06:203 INFO  [node1] Servers in cluster: [node1] [OHazelcastPlugin]
2017-07-11 13:07:06:207 INFO  [node1] Opening database 'db_development'... [OHazelcastPlugin]
2017-07-11 13:07:06:209 INFO  [node1] Loaded configuration for database 'db_development' from disk: /home/ubuntu/data/databases/db_development/distributed-config.json [ODistributedStorage]
2017-07-11 13:07:06:222 INFO  [node1] Distributed servers status (*=current @=lockmgr[node1]):

+-----------+--------+---------+-----+---------+-----------------+-----------------+-----------------------+
|Name       |Status  |Databases|Conns|StartedOn|Binary           |HTTP             |UsedMemory             |
+-----------+--------+---------+-----+---------+-----------------+-----------------+-----------------------+
|node1(*)(@)|STARTING|         |1    |13:07:00 |10.1.1.1:2424    |10.1.1.1:2480    |355.14MB/3.83GB (9.05%)|
+-----------+--------+---------+-----+---------+-----------------+-----------------+-----------------------+
 [OHazelcastPlugin]
2017-07-11 13:07:06:224 INFO  [node1] Current node started as MASTER for database 'db_development' [OHazelcastPlugin$6]$ANSI{green {db=db_development}} Exception `6880A525` in storage `db_development`
com.orientechnologies.orient.core.exception.OStorageException: Cannot open local storage '/home/ubuntu/data/databases/db_development' with mode=rw
	DB name="db_development"
	at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.open(OAbstractPaginatedStorage.java:299)
	at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.open(ODatabaseDocumentTx.java:258)
	at com.orientechnologies.orient.server.OServer.openDatabaseBypassingSecurity(OServer.java:980)
	at com.orientechnologies.orient.server.OServer.openDatabase(OServer.java:947)
	at com.orientechnologies.orient.server.OServer.openDatabase(OServer.java:932)
	at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.reassignClustersOwnership(ODistributedAbstractPlugin.java:733)
	at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin$6.call(OHazelcastPlugin.java:740)
	at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin$6.call(OHazelcastPlugin.java:727)
	at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.executeInDistributedDatabaseLock(ODistributedAbstractPlugin.java:1649)
	at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.loadLocalDatabases(OHazelcastPlugin.java:727)
	at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.startup(OHazelcastPlugin.java:254)
	at com.orientechnologies.orient.server.OServer.registerPlugins(OServer.java:1242)
	at com.orientechnologies.orient.server.OServer.activate(OServer.java:397)
	at com.orientechnologies.orient.server.OServerMain$1.run(OServerMain.java:46)
Caused by: com.orientechnologies.orient.core.exception.OStorageException: Can not open storage it is acquired by other process
	DB name="db_development"
	at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedStorageDirtyFlag.lockFile(OPaginatedStorageDirtyFlag.java:96)
	at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedStorageDirtyFlag.open(OPaginatedStorageDirtyFlag.java:129)
	at com.orientechnologies.orient.core.storage.impl.local.paginated.OLocalPaginatedStorage.preOpenSteps(OLocalPaginatedStorage.java:395)
	at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.open(OAbstractPaginatedStorage.java:250)
	... 13 more
[node1] Error on starting distributed plugin
com.orientechnologies.orient.core.exception.OStorageException: Cannot open local storage '/home/ubuntu/data/databases/db_development' with mode=rw
	DB name="db_development"
	at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.open(OAbstractPaginatedStorage.java:299)
	at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.open(ODatabaseDocumentTx.java:258)
	at com.orientechnologies.orient.server.OServer.openDatabaseBypassingSecurity(OServer.java:980)
	at com.orientechnologies.orient.server.OServer.openDatabase(OServer.java:947)
	at com.orientechnologies.orient.server.OServer.openDatabase(OServer.java:932)
	at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.reassignClustersOwnership(ODistributedAbstractPlugin.java:733)
	at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin$6.call(OHazelcastPlugin.java:740)
	at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin$6.call(OHazelcastPlugin.java:727)
	at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.executeInDistributedDatabaseLock(ODistributedAbstractPlugin.java:1649)
	at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.loadLocalDatabases(OHazelcastPlugin.java:727)
	at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.startup(OHazelcastPlugin.java:254)
	at com.orientechnologies.orient.server.OServer.registerPlugins(OServer.java:1242)
	at com.orientechnologies.orient.server.OServer.activate(OServer.java:397)
	at com.orientechnologies.orient.server.OServerMain$1.run(OServerMain.java:46)
Caused by: com.orientechnologies.orient.core.exception.OStorageException: Can not open storage it is acquired by other process
	DB name="db_development"
	at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedStorageDirtyFlag.lockFile(OPaginatedStorageDirtyFlag.java:96)
	at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedStorageDirtyFlag.open(OPaginatedStorageDirtyFlag.java:129)
	at com.orientechnologies.orient.core.storage.impl.local.paginated.OLocalPaginatedStorage.preOpenSteps(OLocalPaginatedStorage.java:395)
	at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.open(OAbstractPaginatedStorage.java:250)
	... 13 more

2017-07-11 13:07:12:363 WARNI Received signal: SIGINT [OSignalHandler]
2017-07-11 13:07:12:364 INFO  [10.1.1.1]:2481 [root] [3.6.5] Running shutdown hook... Current state: ACTIVE [Node]
2017-07-11 13:07:12:364 INFO  [10.1.1.1]:2481 [root] [3.6.5] Address[10.1.1.1]:2481 is SHUTTING_DOWN [LifecycleService]
2017-07-11 13:07:12:364 WARNI [10.1.1.1]:2481 [root] [3.6.5] Terminating forcefully... [Node]
2017-07-11 13:07:12:364 INFO  [10.1.1.1]:2481 [root] [3.6.5] Shutting down connection manager... [Node]

Fixed issue #5012.

lvca · 2017-07-13T08:02:05Z

About the original error, I found a bug with non-tx locks. Fixed in 2.2.x branch. About the other error, during the installation of the database, something went wrong, but it should be in the logs. Do you still have them?

nishantkumar1292 · 2017-07-13T09:46:19Z

This was the latest log I had which was related to the issue from the logs folder. Are database installation logs stored separately? If no, then I might not have it. Any way fix this issue.. @lvca . We are using Ruby and active-orient to enter data and hitting other queries into OrientDB.
Moreover for the original locks error, do I have to update my OrientDB or will it be released with the next 2.2.x version? We have to use this in production so cant use OrientDB from development branch.

nishantkumar1292 · 2017-07-17T09:00:12Z

Hi @lvca ...any updates how do we resolve the ODistributedRecordLockedException bug on our end?

lvca · 2017-07-18T16:09:12Z

@nishantkumar1292 this bug has been fixed. You can try with last 2.2.x snapshot.

decebal · 2017-10-31T12:44:50Z

I still get this in 2.2.29. I am more than happy to conclude my investigation by switching to saving all my edges in transactions. It looks like in my case it's the right solution to get around this one.

Just wondering what other people think of this, should I wait for this bug to be fix or make sure that the codebase uses transations whenever I save my edges ?

decebal · 2017-10-31T16:55:36Z

Nope, sorry won't work.
I can't of want to blame this commit for all my problems:
07c48dc

tglman · 2017-11-15T18:05:50Z

hi @decebal,

This commit refer to 3.0.0 SNAPSHOT, you don't have this change in 2.2.29.

Regards

lvca added the question label Oct 7, 2015

lvca self-assigned this Oct 7, 2015

lvca added the waiting reply label Jun 9, 2017

lvca added a commit that referenced this issue Jul 13, 2017

HA: check expiration of individual locks (outside Tis)

cd53b84

Fixed issue #5012.

santo-it added this to the 2.2.24 milestone Jul 19, 2017

lvca modified the milestones: 2.2.25, 2.2.24 Aug 2, 2017

lvca closed this as completed Aug 2, 2017

decebal mentioned this issue Oct 23, 2017

ODistributedRecordLockedException or ONeedRetryException not dealt with mogui/pyorient#258

Closed

decebal mentioned this issue Oct 31, 2017

ODistributedRecordLockedException when saving multiple edges between 2 vertexes in a distributed environement #7856

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Getting ODistributedRecordLockedException creating edge in distributed mode #5012

Getting ODistributedRecordLockedException creating edge in distributed mode #5012

stefomak commented Sep 24, 2015

stefomak commented Sep 24, 2015

lvca commented Oct 7, 2015

stefomak commented Oct 7, 2015

stefomak commented Oct 16, 2015

jack18jack commented Mar 8, 2016

nishantkumar1292 commented Jun 9, 2017

lvca commented Jun 9, 2017

nishantkumar1292 commented Jun 9, 2017

lvca commented Jun 10, 2017

nishantkumar1292 commented Jun 13, 2017 •

edited

Loading

lvca commented Jun 13, 2017

nishantkumar1292 commented Jun 22, 2017

lvca commented Jun 22, 2017

nishantkumar1292 commented Jun 23, 2017

roots-ai commented Jun 30, 2017 •

edited

Loading

lvca commented Jun 30, 2017

nishantkumar1292 commented Jul 3, 2017

roots-ai commented Jul 10, 2017

lvca commented Jul 10, 2017

nishantkumar1292 commented Jul 10, 2017 •

edited

Loading

roots-ai commented Jul 11, 2017

nishantkumar1292 commented Jul 12, 2017

lvca commented Jul 12, 2017

lvca commented Jul 12, 2017

lvca commented Jul 12, 2017

nishantkumar1292 commented Jul 12, 2017

lvca commented Jul 13, 2017

nishantkumar1292 commented Jul 13, 2017 •

edited

Loading

nishantkumar1292 commented Jul 17, 2017

lvca commented Jul 18, 2017

decebal commented Oct 31, 2017 •

edited

Loading

decebal commented Oct 31, 2017

tglman commented Nov 15, 2017

Getting ODistributedRecordLockedException creating edge in distributed mode #5012

Getting ODistributedRecordLockedException creating edge in distributed mode #5012

Comments

stefomak commented Sep 24, 2015

stefomak commented Sep 24, 2015

lvca commented Oct 7, 2015

stefomak commented Oct 7, 2015

stefomak commented Oct 16, 2015

jack18jack commented Mar 8, 2016

nishantkumar1292 commented Jun 9, 2017

lvca commented Jun 9, 2017

nishantkumar1292 commented Jun 9, 2017

lvca commented Jun 10, 2017

nishantkumar1292 commented Jun 13, 2017 • edited Loading

lvca commented Jun 13, 2017

nishantkumar1292 commented Jun 22, 2017

lvca commented Jun 22, 2017

nishantkumar1292 commented Jun 23, 2017

roots-ai commented Jun 30, 2017 • edited Loading

lvca commented Jun 30, 2017

nishantkumar1292 commented Jul 3, 2017

roots-ai commented Jul 10, 2017

lvca commented Jul 10, 2017

nishantkumar1292 commented Jul 10, 2017 • edited Loading

roots-ai commented Jul 11, 2017

nishantkumar1292 commented Jul 12, 2017

lvca commented Jul 12, 2017

lvca commented Jul 12, 2017

lvca commented Jul 12, 2017

nishantkumar1292 commented Jul 12, 2017

lvca commented Jul 13, 2017

nishantkumar1292 commented Jul 13, 2017 • edited Loading

nishantkumar1292 commented Jul 17, 2017

lvca commented Jul 18, 2017

decebal commented Oct 31, 2017 • edited Loading

decebal commented Oct 31, 2017

tglman commented Nov 15, 2017

nishantkumar1292 commented Jun 13, 2017 •

edited

Loading

roots-ai commented Jun 30, 2017 •

edited

Loading

nishantkumar1292 commented Jul 10, 2017 •

edited

Loading

nishantkumar1292 commented Jul 13, 2017 •

edited

Loading

decebal commented Oct 31, 2017 •

edited

Loading