distributed: How to relieve "timeout on waiting for synchronous responses from nodes" problem?

[efenzha]

OrientDB Version: orientdb-3.0.21

Java Version: 1.8.0_152

OS: linux

Expected behavior

write to clusters successfully

Actual behavior

lots of "timeout on waiting for synchronous responses from nodes" warning
writing to database is very slow.

Steps to reproduce

1. 3 nodes in distributed mode
2. using Spark to add vertices and edges into database with high rate
3. then lots of "timeout on waiting for synchronous responses from nodes" warning appears on two nodes, and finally Spark crashes because of writing exception.

com.orientechnologies.orient.core.exception.ODatabaseException: Error during saving of record with rid #-1:-1
DB name="liveevents"
at com.orientechnologies.orient.core.tx.OTransactionNoTx.saveRecord(OTransactionNoTx.java:206)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentAbstract.saveInternal(ODatabaseDocumentAbstract.java:2132)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentAbstract.save(ODatabaseDocumentAbstract.java:2097)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentAbstract.save(ODatabaseDocumentAbstract.java:138)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:662)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:66)
at com.orientechnologies.orient.core.record.impl.ODocument.save(ODocument.java:2169)
at com.orientechnologies.orient.core.record.impl.ODocument.save(ODocument.java:2160)
at com.tinkerpop.blueprints.impls.orient.OrientElement.save(OrientElement.java:321)
at com.tinkerpop.blueprints.impls.orient.OrientElement.save(OrientElement.java:304)
at com.tinkerpop.blueprints.impls.orient.OrientBaseGraph.addVertex(OrientBaseGraph.java:605)

2019-06-27 15:18:54:496 WARNI Reached limit of retry for commit tx:0.1529452 forcing database re-install [ODatabaseDocumentDistributed]
2019-06-27 15:18:54:496 WARNI Reached limit of retry for commit tx:0.1529484 forcing database re-install [ODatabaseDocumentDistributed]
2019-06-27 15:18:54:496 WARNI Reached limit of retry for commit tx:0.1529488 forcing database re-install [ODatabaseDocumentDistributed]
2019-06-27 15:19:03:068 WARNI [7889] Timeout (9000ms) on waiting for synchronous responses from nodes=[7888, 7887, 7889] responsesSoFar=[7889] request=(id=2.2186578 task=TxPhase1 user=#5:0) [ODistributedDatabaseImpl]
2019-06-27 15:19:03:122 WARNI [7889] Timeout (9000ms) on waiting for synchronous responses from nodes=[7888, 7887, 7889] responsesSoFar=[7889] request=(id=2.2186620 task=TxPhase1 user=#5:0) [ODistributedDatabaseImpl]
2019-06-27 15:19:12:148 WARNI [7889] Timeout (9000ms) on waiting for synchronous responses from nodes=[7888, 7887, 7889] responsesSoFar=[7889] request=(id=2.2186669 task=TxPhase1 user=#5:0) [ODistributedDatabaseImpl]
2019-06-27 15:19:21:442 WARNI [7889] Timeout (9000ms) on waiting for synchronous responses from nodes=[7888, 7887, 7889] responsesSoFar=[7889] request=(id=2.2186688 task=TxPhase1 user=#5:0) [ODistributedDatabaseImpl]
2019-06-27 15:19:27:324 WARNI [7889] Timeout (30000ms) on waiting for synchronous responses from nodes=[7888, 7887] responsesSoFar=[7887] request=(id=2.2186640 task=upd_db_status) [ODistributedDatabaseImpl]
2019-06-27 15:19:30:099 WARNI [7889] Timeout (9000ms) on waiting for synchronous responses from nodes=[7888, 7887, 7889] responsesSoFar=[7889] request=(id=2.2186692 task=TxPhase1 user=#5:0) [ODistributedDatabaseImpl]