Timeout on waiting for synchronous responses From other nodes

[hxgxs1]

OrientDB Version: 3.0.13

Java Version: 8

OS: Debian 9

Expected behavior

We are using orientDB in distributed mode. If we create/update edges on any of the masters it should get persisted /updated without any exceptions or warnings.
Cluster size: 3 (master, master, replica)

Actual behavior

when we create/update edges, we get following error:

2019-01-28 16:58:20:877 WARNI [node1] Timeout (1213ms) on waiting for synchronous responses from nodes=[node2, node1, node3] responsesSoFar=[node1, node1] request=(id=0.116757 task=TxPhase1) [ODistributedDatabaseImpl]

Steps to reproduce

Any Insert statement on the master is causing this issue.

[cschockaert]

+1 got exactly the same problem with odb 3.3.18 here
never got this problem in odb latest 2.X version

[jonsalvas]

+1. Same issue in odb 3.0.18. Retry loop does not solve the issue either. Occurs even with high timeouts up to 14 s

2019-08-03 13:00:38:619 WARNI [orientdb-0] Timeout (40001ms) on waiting for synchronous responses from nodes=[orientdb-2, orientdb-1] responsesSoFar=[orientdb-1] request=(id=0.20026 task=upd_db_status) [ODistributedDatabaseImpl]

[ygyg70]

I am getting the same exception when starting a node in a cluster (OrientDB 3.0.23):

Timeout (63696ms) on waiting for synchronous responses from nodes=[node_4] responsesSoFar=[] request=(id=1.4 task=copy_db_chunk) [ODistributedDatabaseImpl]Exception 7A575303 in storage plocal:C:\db_folder\databases/dn_name: 3.0.23 - Veloce (build 6f15b98, branch 3.0.x)
java.io.EOFException: Unexpected end of ZLIB input stream
at java.base/java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:245)
at java.base/java.util.zip.InflaterInputStream.read(InflaterInputStream.java:159)
at java.base/java.util.zip.ZipInputStream.read(ZipInputStream.java:195)
at com.orientechnologies.common.io.OIOUtils.copyStream(OIOUtils.java:205)
at com.orientechnologies.orient.core.compression.impl.OZIPCompressionUtil.extractFile(OZIPCompressionUtil.java:97)
at com.orientechnologies.orient.core.compression.impl.OZIPCompressionUtil.uncompressDirectory(OZIPCompressionUtil.java:83)
at com.orientechnologies.orient.core.storage.disk.OLocalPaginatedStorage.restore(OLocalPaginatedStorage.java:294)
at com.orientechnologies.orient.core.db.OrientDBEmbedded.restore(OrientDBEmbedded.java:418)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin$6.call(ODistributedAbstractPlugin.java:1991)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin$6.call(ODistributedAbstractPlugin.java:1930)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.executeInDistributedDatabaseLock(ODistributedAbstractPlugin.java:1770)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.installDatabaseOnLocalNode(ODistributedAbstractPlugin.java:1930)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.installDatabaseFromNetwork(ODistributedAbstractPlugin.java:1597)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.requestDatabaseFullSync(ODistributedAbstractPlugin.java:1418)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.requestFullDatabase(ODistributedAbstractPlugin.java:1100)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin$3.call(ODistributedAbstractPlugin.java:997)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin$3.call(ODistributedAbstractPlugin.java:948)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.executeInDistributedDatabaseLock(ODistributedAbstractPlugin.java:1770)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.installDatabase(ODistributedAbstractPlugin.java:947)
at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.installNewDatabasesFromCluster(OHazelcastPlugin.java:1439)
at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.startup(OHazelcastPlugin.java:300)
at com.orientechnologies.orient.server.OServer.registerPlugins(OServer.java:1194)
at com.orientechnologies.orient.server.OServer.activate(OServer.java:469)

See also #8994

[andersandren]

Same on 3.0.30:
2020-04-02 14:26:13:309 WARNI [EC2AMAZ-D1MVF49] Timeout (20050ms) on waiting for synchronous responses from nodes=[node2, CLUSTER2] responsesSoFar=[CLUSTER2] request=(id=0.12 task=gossip timestamp: 1585830353259 lockManagerServer: node2) [ODistributedDatabaseImpl]

[mehmetaliersoy]

Hei All;

I'm using Orientdb 3.0.28 with 2 cluster and facing similar issue. It prevents writing and our apps throwing Out of Memory error and stops finally.

My default-distributed-db-config.json:

{
"replication": true,
"autoDeploy": true,
"hotAlignment": false,
"executionMode": "undefined",
"readQuorum": 1,
"writeQuorum": "majority",
"readYourWrites": true,
"failureAvailableNodesLessQuorum": false,
"newNodeStrategy": "dynamic",
"servers": {
"": "master"
},
"clusters": {
"internal": {
},
"": {
"servers": ["<NEW_NODE>"]
}
}
}

Any update on it?

OrientDB_Error_Details.txt