2020-06-17 09:51:14:813 INFO Limit of open files for disk cache will be set to 64000. [OEngineLocalPaginated] 2020-06-17 09:51:14:835 INFO Loading configuration from: /app/orientdb/config/orientdb-server-config.xml... [OServerConfigurationLoaderXml] 2020-06-17 09:51:15:020 INFO OrientDB Server v3.0.31 - Veloce (build 2a8412458048d53cd1e1a3544d19b87c190c2d73, branch 3.0.x) is starting up... [OServer] 2020-06-17 09:51:15:038 INFO OrientDB config DISKCACHE=64,000MB [orientechnologies] 2020-06-17 09:51:15:113 INFO System is started under an effective user : `orientdb` [OEngineLocalPaginated] 2020-06-17 09:51:15:113 INFO Allocation of 972800 pages. [OEngineLocalPaginated] 2020-06-17 09:51:18:343 INFO WAL maximum segment size is set to 6,144 MB [OrientDBDistributed] 2020-06-17 09:51:18:344 INFO Databases directory: /app/orientdb/databases [OServer] 2020-06-17 09:51:18:412 INFO Direct IO for WAL located in /app/orientdb/databases/OSystem is allowed with block size 4096 bytes. [OCASDiskWriteAheadLog] 2020-06-17 09:51:18:412 INFO Page size for WAL located in /app/orientdb/databases/OSystem is set to 4096 bytes. [OCASDiskWriteAheadLog] 2020-06-17 09:51:19:066 INFO Storage 'plocal:/app/orientdb/databases/OSystem' is opened under OrientDB distribution : 3.0.31 - Veloce (build 2a8412458048d53cd1e1a3544d19b87c190c2d73, branch 3.0.x) [OLocalPaginatedStorage] 2020-06-17 09:51:19:435 INFO Listening binary connections on 0.0.0.0:2424 (protocol v.37, socket=default) [OServerNetworkListener] 2020-06-17 09:51:19:438 INFO Listening http connections on 0.0.0.0:2480 (protocol v.10, socket=default) [OServerNetworkListener] 2020-06-17 09:51:19:445 INFO Installing dynamic plugin 'orientdb-neo4j-importer-plugin-3.0.31-dist.jar'... [OServerPluginManager] 2020-06-17 09:51:19:449 INFO Installing dynamic plugin 'orientdb-etl-3.0.31.jar'... [OServerPluginManager] 2020-06-17 09:51:19:452 INFO Installing dynamic plugin 'orientdb-teleporter-3.0.31.jar'... [OServerPluginManager] 2020-06-17 09:51:19:456 INFO Installing dynamic plugin 'orientdb-studio-3.0.31.zip'... [OServerPluginManager] 2020-06-17 09:51:19:461 INFO ODefaultPasswordAuthenticator is active [ODefaultPasswordAuthenticator] 2020-06-17 09:51:19:462 INFO OServerConfigAuthenticator is active [OServerConfigAuthenticator] 2020-06-17 09:51:19:462 INFO OSystemUserAuthenticator is active [OSystemUserAuthenticator] 2020-06-17 09:51:19:751 INFO Configuring Hazelcast from '/app/orientdb/config/hazelcast.xml'. [FileSystemXmlConfig] 2020-06-17 09:51:19:807 WARNI Name of the hazelcast schema location is incorrect, using default [AbstractXmlConfigHelper] 2020-06-17 09:51:20:042 INFO [LOCAL] [orientdb] [3.10.6] Interfaces is disabled, trying to pick one address from TCP-IP config addresses: [172.16.5.72, 172.16.5.71, 172.16.5.74, 172.16.5.73, 172.16.5.76, 172.16.5.75] [AddressPicker] 2020-06-17 09:51:20:042 INFO [LOCAL] [orientdb] [3.10.6] Prefer IPv4 stack is true. [AddressPicker] 2020-06-17 09:51:20:047 INFO [LOCAL] [orientdb] [3.10.6] Picked [172.16.5.71]:2434, using socket ServerSocket[addr=/0:0:0:0:0:0:0:0,localport=2434], bind any local is true [AddressPicker] 2020-06-17 09:51:20:058 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Hazelcast 3.10.6 (20181010 - fdcf6d6) starting at [172.16.5.71]:2434 [system] 2020-06-17 09:51:20:058 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Copyright (c) 2008-2018, Hazelcast, Inc. All Rights Reserved. [system] 2020-06-17 09:51:20:059 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Configured Hazelcast Serialization version: 1 [system] 2020-06-17 09:51:20:059 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] A non-empty group password is configured for the Hazelcast member. Starting with Hazelcast version 3.8.2, members with the same group name, but with different group passwords (that do not use authentication) form a cluster. The group password configuration will be removed completely in a future release. [Node] 2020-06-17 09:51:20:222 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Backpressure is disabled [BackpressureRegulator] 2020-06-17 09:51:20:232 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Running with 2 response threads [InboundResponseHandlerSupplier] 2020-06-17 09:51:20:637 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Creating TcpIpJoiner [Node] 2020-06-17 09:51:20:755 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Starting 1 partition threads and 2 generic threads (1 dedicated for priority tasks) [OperationExecutorImpl] 2020-06-17 09:51:20:757 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Diagnostics disabled. To enable add -Dhazelcast.diagnostics.enabled=true to the JVM arguments. [Diagnostics] 2020-06-17 09:51:20:771 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] [172.16.5.71]:2434 is STARTING [LifecycleService] 2020-06-17 09:51:20:789 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Connecting to /172.16.5.74:2435, timeout: 0, bind-any: true [TcpIpConnector] 2020-06-17 09:51:20:789 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Could not connect to: /172.16.5.74:2435. Reason: SocketException[Connection refused to address /172.16.5.74:2435] [TcpIpConnector] 2020-06-17 09:51:20:790 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] [172.16.5.74]:2435 is added to the blacklist. [TcpIpJoiner] 2020-06-17 09:51:20:791 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Connecting to /172.16.5.73:2434, timeout: 0, bind-any: true [TcpIpConnector] 2020-06-17 09:51:20:792 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Connecting to /172.16.5.75:2436, timeout: 0, bind-any: true [TcpIpConnector] 2020-06-17 09:51:20:792 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Connecting to /172.16.5.74:2434, timeout: 0, bind-any: true [TcpIpConnector] 2020-06-17 09:51:20:792 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Connecting to /172.16.5.76:2436, timeout: 0, bind-any: true [TcpIpConnector] 2020-06-17 09:51:20:792 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Connecting to /172.16.5.75:2435, timeout: 0, bind-any: true [TcpIpConnector] 2020-06-17 09:51:20:793 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Connecting to /172.16.5.72:2435, timeout: 0, bind-any: true [TcpIpConnector] 2020-06-17 09:51:20:793 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Could not connect to: /172.16.5.75:2435. Reason: SocketException[Connection refused to address /172.16.5.75:2435] [TcpIpConnector] 2020-06-17 09:51:20:793 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Could not connect to: /172.16.5.74:2434. Reason: SocketException[Connection refused to address /172.16.5.74:2434] [TcpIpConnector] 2020-06-17 09:51:20:794 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Could not connect to: /172.16.5.73:2434. Reason: SocketException[Connection refused to address /172.16.5.73:2434] [TcpIpConnector] 2020-06-17 09:51:20:795 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] [172.16.5.73]:2434 is added to the blacklist. [TcpIpJoiner] 2020-06-17 09:51:20:795 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Connecting to /172.16.5.73:2436, timeout: 0, bind-any: true [TcpIpConnector] 2020-06-17 09:51:20:796 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Connecting to /172.16.5.74:2436, timeout: 0, bind-any: true [TcpIpConnector] 2020-06-17 09:51:20:797 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Connecting to /172.16.5.75:2434, timeout: 0, bind-any: true [TcpIpConnector] 2020-06-17 09:51:20:797 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Could not connect to: /172.16.5.75:2436. Reason: SocketException[Connection refused to address /172.16.5.75:2436] [TcpIpConnector] 2020-06-17 09:51:20:798 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Connecting to /172.16.5.76:2435, timeout: 0, bind-any: true [TcpIpConnector] 2020-06-17 09:51:20:798 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Could not connect to: /172.16.5.73:2436. Reason: SocketException[Connection refused to address /172.16.5.73:2436] [TcpIpConnector] 2020-06-17 09:51:20:798 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Connecting to /172.16.5.72:2434, timeout: 0, bind-any: true [TcpIpConnector] 2020-06-17 09:51:20:799 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Could not connect to: /172.16.5.72:2435. Reason: SocketException[Connection refused to address /172.16.5.72:2435] [TcpIpConnector] 2020-06-17 09:51:20:799 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] [172.16.5.75]:2435 is added to the blacklist. [TcpIpJoiner] 2020-06-17 09:51:20:799 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Connecting to /172.16.5.71:2435, timeout: 0, bind-any: true [TcpIpConnector] 2020-06-17 09:51:20:799 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] [172.16.5.72]:2435 is added to the blacklist. [TcpIpJoiner] 2020-06-17 09:51:20:799 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Could not connect to: /172.16.5.76:2435. Reason: SocketException[Connection refused to address /172.16.5.76:2435] [TcpIpConnector] 2020-06-17 09:51:20:799 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Could not connect to: /172.16.5.76:2436. Reason: SocketException[Connection refused to address /172.16.5.76:2436] [TcpIpConnector] 2020-06-17 09:51:20:801 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] [172.16.5.75]:2436 is added to the blacklist. [TcpIpJoiner] 2020-06-17 09:51:20:802 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Connecting to /172.16.5.72:2436, timeout: 0, bind-any: true [TcpIpConnector] 2020-06-17 09:51:20:802 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Could not connect to: /172.16.5.74:2436. Reason: SocketException[Connection refused to address /172.16.5.74:2436] [TcpIpConnector] 2020-06-17 09:51:20:803 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Connecting to /172.16.5.73:2435, timeout: 0, bind-any: true [TcpIpConnector] 2020-06-17 09:51:20:803 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] [172.16.5.74]:2434 is added to the blacklist. [TcpIpJoiner] 2020-06-17 09:51:20:803 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Connecting to /172.16.5.71:2436, timeout: 0, bind-any: true [TcpIpConnector] 2020-06-17 09:51:20:803 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] [172.16.5.74]:2436 is added to the blacklist. [TcpIpJoiner] 2020-06-17 09:51:20:804 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] [172.16.5.76]:2435 is added to the blacklist. [TcpIpJoiner] 2020-06-17 09:51:20:804 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] [172.16.5.76]:2436 is added to the blacklist. [TcpIpJoiner] 2020-06-17 09:51:20:804 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Could not connect to: /172.16.5.72:2434. Reason: SocketException[Connection refused to address /172.16.5.72:2434] [TcpIpConnector] 2020-06-17 09:51:20:804 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Could not connect to: /172.16.5.72:2436. Reason: SocketException[Connection refused to address /172.16.5.72:2436] [TcpIpConnector] 2020-06-17 09:51:20:807 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Could not connect to: /172.16.5.71:2435. Reason: SocketException[Connection refused to address /172.16.5.71:2435] [TcpIpConnector] 2020-06-17 09:51:20:807 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Connecting to /172.16.5.76:2434, timeout: 0, bind-any: true [TcpIpConnector] 2020-06-17 09:51:20:807 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] [172.16.5.72]:2436 is added to the blacklist. [TcpIpJoiner] 2020-06-17 09:51:20:807 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Could not connect to: /172.16.5.75:2434. Reason: SocketException[Connection refused to address /172.16.5.75:2434] [TcpIpConnector] 2020-06-17 09:51:20:807 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Could not connect to: /172.16.5.73:2435. Reason: SocketException[Connection refused to address /172.16.5.73:2435] [TcpIpConnector] 2020-06-17 09:51:20:808 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] [172.16.5.72]:2434 is added to the blacklist. [TcpIpJoiner] 2020-06-17 09:51:20:808 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] [172.16.5.73]:2436 is added to the blacklist. [TcpIpJoiner] 2020-06-17 09:51:20:808 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] [172.16.5.71]:2435 is added to the blacklist. [TcpIpJoiner] 2020-06-17 09:51:20:809 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] [172.16.5.73]:2435 is added to the blacklist. [TcpIpJoiner] 2020-06-17 09:51:20:809 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Could not connect to: /172.16.5.71:2436. Reason: SocketException[Connection refused to address /172.16.5.71:2436] [TcpIpConnector] 2020-06-17 09:51:20:809 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Could not connect to: /172.16.5.76:2434. Reason: SocketException[Connection refused to address /172.16.5.76:2434] [TcpIpConnector] 2020-06-17 09:51:20:810 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] [172.16.5.75]:2434 is added to the blacklist. [TcpIpJoiner] 2020-06-17 09:51:20:810 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] [172.16.5.76]:2434 is added to the blacklist. [TcpIpJoiner] 2020-06-17 09:51:20:810 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] [172.16.5.71]:2436 is added to the blacklist. [TcpIpJoiner] 2020-06-17 09:51:21:797 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Cluster version set to 3.10 [system] 2020-06-17 09:51:21:798 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Members {size:1, ver:1} [ Member [172.16.5.71]:2434 - 111d390c-f6c1-48c3-890b-a004ae134975 this ] [ClusterService] 2020-06-17 09:51:21:816 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] [172.16.5.71]:2434 is STARTED [LifecycleService] 2020-06-17 09:51:21:817 INFO Starting distributed server 'ograph01-5-71' (hzID=111d390c-f6c1-48c3-890b-a004ae134975)... [OHazelcastPlugin] 2020-06-17 09:51:21:820 INFO Distributed cluster time=Wed Jun 17 09:51:21 ICT 2020 (delta from local node=0)... [OHazelcastPlugin] 2020-06-17 09:51:21:855 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Initializing cluster partition table arrangement... [PartitionStateManager] 2020-06-17 09:51:21:938 INFO [ograph01-5-71] Registered local server with nodeId=0 [OHazelcastPlugin] 2020-06-17 09:51:21:953 INFO Distributed Lock Manager server is 'ograph01-5-71' [OHazelcastPlugin] 2020-06-17 09:51:21:960 INFO [ograph01-5-71] Loaded configuration for database 'OSystem' from disk: /app/orientdb/config/default-distributed-db-config.json [ODistributedStorage] 2020-06-17 09:51:22:032 INFO [ograph01-5-71] Servers in cluster: [ograph01-5-71] [OHazelcastPlugin] 2020-06-17 09:51:22:055 INFO [ograph01-5-71] Opening database 'web'... [OHazelcastPlugin] 2020-06-17 09:51:22:057 INFO [ograph01-5-71] Loaded configuration for database 'web' from disk: /app/orientdb/databases/web/distributed-config.json [ODistributedStorage] 2020-06-17 09:51:22:217 INFO [ograph01-5-71] Distributed servers status (*=current @=lockmgr[ograph01-5-71]): +-------------------+--------+---------+-----+---------+----------------+----------------+------------------------+ |Name |Status |Databases|Conns|StartedOn|Binary |HTTP |UsedMemory | +-------------------+--------+---------+-----+---------+----------------+----------------+------------------------+ |ograph01-5-71(*)(@)|STARTING| |0 |09:51:19 |172.16.5.71:2424|172.16.5.71:2480|478.51MB/64.00GB (0.73%)| +-------------------+--------+---------+-----+---------+----------------+----------------+------------------------+ [OHazelcastPlugin] 2020-06-17 09:51:22:233 INFO [ograph01-5-71] Current node started as MASTER for database 'web' [OHazelcastPlugin$4] 2020-06-17 09:51:22:343 INFO Direct IO for WAL located in /app/orientdb/databases/web is allowed with block size 4096 bytes. [OCASDiskWriteAheadLog] 2020-06-17 09:51:22:343 INFO Page size for WAL located in /app/orientdb/databases/web is set to 4096 bytes. [OCASDiskWriteAheadLog] 2020-06-17 09:51:22:924 INFO Storage 'plocal:/app/orientdb/databases/web' is opened under OrientDB distribution : 3.0.31 - Veloce (build 2a8412458048d53cd1e1a3544d19b87c190c2d73, branch 3.0.x) [OLocalPaginatedStorage] 2020-06-17 09:51:23:192 WARNI [ograph01-5-71] Node 'ograph01-5-71' is NOT_AVAILABLE, the command 'sql.select from OUser where name = ? limit 1' against database 'web' will be executed only on local server with the possibility to have partial result [ODistributedStorage] 2020-06-17 09:51:23:198 INFO [ograph01-5-71] Reassigning ownership of clusters for database web... [OHazelcastPlugin] 2020-06-17 09:51:23:200 INFO [ograph01-5-71] Reassignment of clusters for database 'web' completed (classes=167) [OHazelcastPlugin] 2020-06-17 09:51:23:205 INFO [ograph01-5-71] Publishing ONLINE status for database ograph01-5-71.web... [ODistributedDatabaseImpl] 2020-06-17 09:51:23:208 INFO [ograph01-5-71] Updated node status to 'ONLINE' [OHazelcastPlugin] 2020-06-17 09:51:23:210 INFO [ograph01-5-71] Distributed servers status (*=current @=lockmgr[ograph01-5-71]): +-------------------+------+-------------------+-----+---------+----------------+----------------+------------------------+ |Name |Status|Databases |Conns|StartedOn|Binary |HTTP |UsedMemory | +-------------------+------+-------------------+-----+---------+----------------+----------------+------------------------+ |ograph01-5-71(*)(@)|ONLINE|web=ONLINE (MASTER)|0 |09:51:19 |172.16.5.71:2424|172.16.5.71:2480|748.09MB/64.00GB (1.14%)| +-------------------+------+-------------------+-----+---------+----------------+----------------+------------------------+ [OHazelcastPlugin] 2020-06-17 09:51:23:212 WARNI Authenticated clients can execute any kind of code into the server by using the following allowed languages: [sql] [OServerSideScriptInterpreter] 2020-06-17 09:51:23:213 INFO OrientDB Studio available at http://172.16.5.71:2480/studio/index.html [OServer] 2020-06-17 09:51:23:213 INFO OrientDB Server is active v3.0.31 - Veloce (build 2a8412458048d53cd1e1a3544d19b87c190c2d73, branch 3.0.x). [OServer] 2020-06-17 09:51:37:313 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Accepting socket connection from /172.16.5.72:50517 [TcpIpAcceptor] 2020-06-17 09:51:37:314 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Established socket connection between /172.16.5.71:2434 and /172.16.5.72:50517 [TcpIpConnectionManager] 2020-06-17 09:51:44:320 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Members {size:2, ver:2} [ Member [172.16.5.71]:2434 - 111d390c-f6c1-48c3-890b-a004ae134975 this Member [172.16.5.72]:2434 - 6909ffd6-153d-49ce-9286-8defd6aff559 ] [ClusterService] 2020-06-17 09:51:44:323 INFO [ograph01-5-71] Added new node id=Member [172.16.5.72]:2434 - 6909ffd6-153d-49ce-9286-8defd6aff559 name=ext:6909ffd6-153d-49ce-9286-8defd6aff559 [OHazelcastPlugin] 2020-06-17 09:51:44:531 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Re-partitioning cluster data... Migration queue size: 271 [MigrationManager] 2020-06-17 09:51:45:432 INFO [ograph01-5-71]<-[ext:6909ffd6-153d-49ce-9286-8defd6aff559] Received updated about registered nodes [OHazelcastPlugin] 2020-06-17 09:51:45:596 INFO [ograph01-5-71]<-[ograph02-5-72] Added node configuration id=Member [172.16.5.72]:2434 - 6909ffd6-153d-49ce-9286-8defd6aff559 name=ograph02-5-72, now 2 nodes are configured [OHazelcastPlugin] 2020-06-17 09:51:45:598 INFO [ograph01-5-71] Distributed servers status (*=current @=lockmgr[ograph01-5-71]): +-------------------+--------+-------------------+-----+---------+----------------+----------------+------------------------+ |Name |Status |Databases |Conns|StartedOn|Binary |HTTP |UsedMemory | +-------------------+--------+-------------------+-----+---------+----------------+----------------+------------------------+ |ograph02-5-72 |STARTING| |1 |09:51:36 |172.16.5.72:2424|172.16.5.72:2480|456.61MB/64.00GB (0.70%)| |ograph01-5-71(*)(@)|ONLINE |web=ONLINE (MASTER)|0 |09:51:19 |172.16.5.71:2424|172.16.5.71:2480|760.75MB/64.00GB (1.16%)| +-------------------+--------+-------------------+-----+---------+----------------+----------------+------------------------+ [OHazelcastPlugin] 2020-06-17 09:51:45:599 INFO [ograph01-5-71] Distributed servers status (*=current @=lockmgr[ograph01-5-71]): +-------------------+------+-------------------+-----+---------+----------------+----------------+------------------------+ |Name |Status|Databases |Conns|StartedOn|Binary |HTTP |UsedMemory | +-------------------+------+-------------------+-----+---------+----------------+----------------+------------------------+ |ograph02-5-72 | | | | | | | | |ograph01-5-71(*)(@)|ONLINE|web=ONLINE (MASTER)|0 |09:51:19 |172.16.5.71:2424|172.16.5.71:2480|760.75MB/64.00GB (1.16%)| +-------------------+------+-------------------+-----+---------+----------------+----------------+------------------------+ [OHazelcastPlugin] 2020-06-17 09:51:45:601 INFO [ograph01-5-71] Distributed servers status (*=current @=lockmgr[ograph01-5-71]): +-------------------+--------+-------------------+-----+---------+----------------+----------------+------------------------+ |Name |Status |Databases |Conns|StartedOn|Binary |HTTP |UsedMemory | +-------------------+--------+-------------------+-----+---------+----------------+----------------+------------------------+ |ograph02-5-72 |STARTING| |0 |09:51:36 |172.16.5.72:2424|172.16.5.72:2480|453.88MB/64.00GB (0.69%)| |ograph01-5-71(*)(@)|ONLINE |web=ONLINE (MASTER)|0 |09:51:19 |172.16.5.71:2424|172.16.5.71:2480|760.75MB/64.00GB (1.16%)| +-------------------+--------+-------------------+-----+---------+----------------+----------------+------------------------+ [OHazelcastPlugin] 2020-06-17 09:51:45:677 INFO [ograph01-5-71] Setting new distributed configuration for database: OSystem (version=1) CLUSTER CONFIGURATION [wQuorum: true] (LEGEND: X = Owner, o = Copy) +--------+-----------+----------+ |CLUSTER |writeQuorum|readQuorum| +--------+-----------+----------+ |* | 1 | 1 | |internal| 1 | 1 | +--------+-----------+----------+ [ODistributedStorage] 2020-06-17 09:51:46:977 INFO [ograph01-5-71] Received new status ograph02-5-72.web=SYNCHRONIZING [OHazelcastPlugin] 2020-06-17 09:51:46:978 INFO [ograph01-5-71]->[ograph02-5-72] Deploying database 'web' with delta of changes... [OSyncDatabaseDeltaTask] 2020-06-17 09:51:46:978 INFO [ograph01-5-71]->[ograph02-5-72] Creating delta backup of database 'web' (startLSN=OLogSequenceNumber{segment=14489, position=22540320}) in directory: /tmp/orientdb/backup_ograph02-5-72_web_server0.zip... [OSyncDatabaseDeltaTask] 2020-06-17 09:51:46:980 INFO Cannot find requested LSN=OLogSequenceNumber{segment=14489, position=22540320} for database sync operation (last available LSN is OLogSequenceNumber{segment=14491, position=8210}) [OLocalPaginatedStorage] 2020-06-17 09:51:47:001 INFO [ograph01-5-71]<-[ograph02-5-72] Received updated status ograph02-5-72.web=ONLINE [OHazelcastPlugin] 2020-06-17 09:51:47:005 INFO [ograph01-5-71]<-[ograph02-5-72] Received updated status ograph02-5-72.web=NOT_AVAILABLE [OHazelcastPlugin] 2020-06-17 09:51:47:027 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] All migration tasks have been completed, queues are empty. [MigrationThread] 2020-06-17 09:51:53:212 INFO [ograph01-5-71] Distributed servers status (*=current @=lockmgr[ograph01-5-71]): +-------------------+------+--------------------------+-----+---------+----------------+----------------+------------------------+ |Name |Status|Databases |Conns|StartedOn|Binary |HTTP |UsedMemory | +-------------------+------+--------------------------+-----+---------+----------------+----------------+------------------------+ |ograph02-5-72 |ONLINE|web=NOT_AVAILABLE (MASTER)|2 |09:51:36 |172.16.5.72:2424|172.16.5.72:2480|691.21MB/64.00GB (1.05%)| |ograph01-5-71(*)(@)|ONLINE|web=ONLINE (MASTER) |0 |09:51:19 |172.16.5.71:2424|172.16.5.71:2480|760.75MB/64.00GB (1.16%)| +-------------------+------+--------------------------+-----+---------+----------------+----------------+------------------------+ [OHazelcastPlugin] 2020-06-17 09:51:57:076 INFO [ograph01-5-71] Received updated status ograph02-5-72.web=SYNCHRONIZING [OHazelcastPlugin] 2020-06-17 09:51:57:096 INFO [ograph01-5-71]->[ograph02-5-72] Deploying database 'web' with delta of changes... [OSyncDatabaseDeltaTask] 2020-06-17 09:51:57:096 INFO [ograph01-5-71]->[ograph02-5-72] Creating delta backup of database 'web' (startLSN=OLogSequenceNumber{segment=14489, position=22540320}) in directory: /tmp/orientdb/backup_ograph02-5-72_web_server0.zip... [OSyncDatabaseDeltaTask] 2020-06-17 09:51:57:096 INFO Cannot find requested LSN=OLogSequenceNumber{segment=14489, position=22540320} for database sync operation (last available LSN is OLogSequenceNumber{segment=14491, position=8210}) [OLocalPaginatedStorage] 2020-06-17 09:51:57:103 INFO [ograph01-5-71]<-[ograph02-5-72] Received updated status ograph02-5-72.web=ONLINE [OHazelcastPlugin] 2020-06-17 09:51:57:104 INFO [ograph01-5-71] Distributed servers status (*=current @=lockmgr[ograph01-5-71]): +-------------------+------+--------------------------+-----+---------+----------------+----------------+------------------------+ |Name |Status|Databases |Conns|StartedOn|Binary |HTTP |UsedMemory | +-------------------+------+--------------------------+-----+---------+----------------+----------------+------------------------+ |ograph02-5-72 |ONLINE|web=SYNCHRONIZING (MASTER)|2 |09:51:36 |172.16.5.72:2424|172.16.5.72:2480|694.75MB/64.00GB (1.06%)| |ograph01-5-71(*)(@)|ONLINE|web=ONLINE (MASTER) |2 |09:51:19 |172.16.5.71:2424|172.16.5.71:2480|699.81MB/64.00GB (1.07%)| +-------------------+------+--------------------------+-----+---------+----------------+----------------+------------------------+ [OHazelcastPlugin] 2020-06-17 09:52:03:212 INFO [ograph01-5-71] Distributed servers status (*=current @=lockmgr[ograph01-5-71]): +-------------------+------+-------------------+-----+---------+----------------+----------------+------------------------+ |Name |Status|Databases |Conns|StartedOn|Binary |HTTP |UsedMemory | +-------------------+------+-------------------+-----+---------+----------------+----------------+------------------------+ |ograph02-5-72 |ONLINE|web=ONLINE (MASTER)|2 |09:51:36 |172.16.5.72:2424|172.16.5.72:2480|694.75MB/64.00GB (1.06%)| |ograph01-5-71(*)(@)|ONLINE|web=ONLINE (MASTER)|2 |09:51:19 |172.16.5.71:2424|172.16.5.71:2480|699.81MB/64.00GB (1.07%)| +-------------------+------+-------------------+-----+---------+----------------+----------------+------------------------+ [OHazelcastPlugin] 2020-06-17 09:52:10:858 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Accepting socket connection from /172.16.5.73:60981 [TcpIpAcceptor] 2020-06-17 09:52:10:858 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Established socket connection between /172.16.5.71:2434 and /172.16.5.73:60981 [TcpIpConnectionManager] 2020-06-17 09:52:17:865 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Members {size:3, ver:3} [ Member [172.16.5.71]:2434 - 111d390c-f6c1-48c3-890b-a004ae134975 this Member [172.16.5.72]:2434 - 6909ffd6-153d-49ce-9286-8defd6aff559 Member [172.16.5.73]:2434 - 6cbb8848-2d8f-44f9-aad9-556bae5e5c55 ] [ClusterService] 2020-06-17 09:52:17:868 INFO [ograph01-5-71] Added new node id=Member [172.16.5.73]:2434 - 6cbb8848-2d8f-44f9-aad9-556bae5e5c55 name=ext:6cbb8848-2d8f-44f9-aad9-556bae5e5c55 [OHazelcastPlugin] 2020-06-17 09:52:18:125 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Re-partitioning cluster data... Migration queue size: 271 [MigrationManager] 2020-06-17 09:52:18:990 INFO [ograph01-5-71]<-[ext:6cbb8848-2d8f-44f9-aad9-556bae5e5c55] Received updated about registered nodes [OHazelcastPlugin] 2020-06-17 09:52:19:205 INFO [ograph01-5-71]<-[ograph03-5-73] Added node configuration id=Member [172.16.5.73]:2434 - 6cbb8848-2d8f-44f9-aad9-556bae5e5c55 name=ograph03-5-73, now 3 nodes are configured [OHazelcastPlugin] 2020-06-17 09:52:19:208 INFO [ograph01-5-71] Distributed servers status (*=current @=lockmgr[ograph01-5-71]): +-------------------+--------+-------------------+-----+---------+----------------+----------------+------------------------+ |Name |Status |Databases |Conns|StartedOn|Binary |HTTP |UsedMemory | +-------------------+--------+-------------------+-----+---------+----------------+----------------+------------------------+ |ograph02-5-72 |ONLINE |web=ONLINE (MASTER)|2 |09:51:36 |172.16.5.72:2424|172.16.5.72:2480|707.79MB/64.00GB (1.08%)| |ograph01-5-71(*)(@)|ONLINE |web=ONLINE (MASTER)|2 |09:51:19 |172.16.5.71:2424|172.16.5.71:2480|707.39MB/64.00GB (1.08%)| |ograph03-5-73 |STARTING| |0 |09:52:09 |172.16.5.73:2424|172.16.5.73:2480|492.30MB/64.00GB (0.75%)| +-------------------+--------+-------------------+-----+---------+----------------+----------------+------------------------+ [OHazelcastPlugin] 2020-06-17 09:52:19:209 INFO [ograph01-5-71] Distributed servers status (*=current @=lockmgr[ograph01-5-71]): +-------------------+------+-------------------+-----+---------+----------------+----------------+------------------------+ |Name |Status|Databases |Conns|StartedOn|Binary |HTTP |UsedMemory | +-------------------+------+-------------------+-----+---------+----------------+----------------+------------------------+ |ograph02-5-72 |ONLINE|web=ONLINE (MASTER)|2 |09:51:36 |172.16.5.72:2424|172.16.5.72:2480|707.79MB/64.00GB (1.08%)| |ograph01-5-71(*)(@)|ONLINE|web=ONLINE (MASTER)|2 |09:51:19 |172.16.5.71:2424|172.16.5.71:2480|707.39MB/64.00GB (1.08%)| |ograph03-5-73 | | | | | | | | +-------------------+------+-------------------+-----+---------+----------------+----------------+------------------------+ [OHazelcastPlugin] 2020-06-17 09:52:19:211 INFO [ograph01-5-71] Distributed servers status (*=current @=lockmgr[ograph01-5-71]): +-------------------+--------+-------------------+-----+---------+----------------+----------------+------------------------+ |Name |Status |Databases |Conns|StartedOn|Binary |HTTP |UsedMemory | +-------------------+--------+-------------------+-----+---------+----------------+----------------+------------------------+ |ograph02-5-72 |ONLINE |web=ONLINE (MASTER)|2 |09:51:36 |172.16.5.72:2424|172.16.5.72:2480|707.79MB/64.00GB (1.08%)| |ograph01-5-71(*)(@)|ONLINE |web=ONLINE (MASTER)|2 |09:51:19 |172.16.5.71:2424|172.16.5.71:2480|707.39MB/64.00GB (1.08%)| |ograph03-5-73 |STARTING| |0 |09:52:09 |172.16.5.73:2424|172.16.5.73:2480|490.40MB/64.00GB (0.75%)| +-------------------+--------+-------------------+-----+---------+----------------+----------------+------------------------+ [OHazelcastPlugin] 2020-06-17 09:52:19:212 INFO [ograph01-5-71]<-[ograph03-5-73] Received new status ograph03-5-73.OSystem=NOT_AVAILABLE [OHazelcastPlugin] 2020-06-17 09:52:20:409 INFO [ograph01-5-71]->[ograph03-5-73] Deploying database 'web' with delta of changes... [OSyncDatabaseDeltaTask] 2020-06-17 09:52:20:409 INFO [ograph01-5-71] Received new status ograph03-5-73.web=SYNCHRONIZING [OHazelcastPlugin] 2020-06-17 09:52:20:409 INFO [ograph01-5-71]->[ograph03-5-73] Creating delta backup of database 'web' (startLSN=OLogSequenceNumber{segment=14489, position=37412896}) in directory: /tmp/orientdb/backup_ograph03-5-73_web_server0.zip... [OSyncDatabaseDeltaTask] 2020-06-17 09:52:20:409 INFO Cannot find requested LSN=OLogSequenceNumber{segment=14489, position=37412896} for database sync operation (last available LSN is OLogSequenceNumber{segment=14491, position=8210}) [OLocalPaginatedStorage] 2020-06-17 09:52:20:416 INFO [ograph01-5-71]<-[ograph03-5-73] Received updated status ograph03-5-73.web=ONLINE [OHazelcastPlugin] 2020-06-17 09:52:20:420 INFO [ograph01-5-71]<-[ograph03-5-73] Received updated status ograph03-5-73.web=NOT_AVAILABLE [OHazelcastPlugin] 2020-06-17 09:52:20:472 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] All migration tasks have been completed, queues are empty. [MigrationThread] 2020-06-17 09:52:23:212 INFO [ograph01-5-71] Distributed servers status (*=current @=lockmgr[ograph01-5-71]): +-------------------+------+--------------------------+-----+---------+----------------+----------------+------------------------+ |Name |Status|Databases |Conns|StartedOn|Binary |HTTP |UsedMemory | +-------------------+------+--------------------------+-----+---------+----------------+----------------+------------------------+ |ograph02-5-72 |ONLINE|web=ONLINE (MASTER) |2 |09:51:36 |172.16.5.72:2424|172.16.5.72:2480|707.79MB/64.00GB (1.08%)| |ograph01-5-71(*)(@)|ONLINE|web=ONLINE (MASTER) |2 |09:51:19 |172.16.5.71:2424|172.16.5.71:2480|707.39MB/64.00GB (1.08%)| |ograph03-5-73 |ONLINE|web=NOT_AVAILABLE (MASTER)|4 |09:52:09 |172.16.5.73:2424|172.16.5.73:2480|712.79MB/64.00GB (1.09%)| +-------------------+------+--------------------------+-----+---------+----------------+----------------+------------------------+ [OHazelcastPlugin] 2020-06-17 09:52:30:539 INFO [ograph01-5-71]->[ograph03-5-73] Deploying database 'web' with delta of changes... [OSyncDatabaseDeltaTask] 2020-06-17 09:52:30:539 INFO [ograph01-5-71]->[ograph03-5-73] Creating delta backup of database 'web' (startLSN=OLogSequenceNumber{segment=14489, position=37412896}) in directory: /tmp/orientdb/backup_ograph03-5-73_web_server0.zip... [OSyncDatabaseDeltaTask] 2020-06-17 09:52:30:539 INFO [ograph01-5-71] Received updated status ograph03-5-73.web=SYNCHRONIZING [OHazelcastPlugin] 2020-06-17 09:52:30:540 INFO Cannot find requested LSN=OLogSequenceNumber{segment=14489, position=37412896} for database sync operation (last available LSN is OLogSequenceNumber{segment=14491, position=8210}) [OLocalPaginatedStorage] 2020-06-17 09:52:30:541 INFO [ograph01-5-71] Distributed servers status (*=current @=lockmgr[ograph01-5-71]): +-------------------+------+--------------------------+-----+---------+----------------+----------------+------------------------+ |Name |Status|Databases |Conns|StartedOn|Binary |HTTP |UsedMemory | +-------------------+------+--------------------------+-----+---------+----------------+----------------+------------------------+ |ograph02-5-72 |ONLINE|web=ONLINE (MASTER) |4 |09:51:36 |172.16.5.72:2424|172.16.5.72:2480|738.93MB/64.00GB (1.13%)| |ograph01-5-71(*)(@)|ONLINE|web=ONLINE (MASTER) |4 |09:51:19 |172.16.5.71:2424|172.16.5.71:2480|757.35MB/64.00GB (1.16%)| |ograph03-5-73 |ONLINE|web=SYNCHRONIZING (MASTER)|4 |09:52:09 |172.16.5.73:2424|172.16.5.73:2480|716.46MB/64.00GB (1.09%)| +-------------------+------+--------------------------+-----+---------+----------------+----------------+------------------------+ [OHazelcastPlugin] 2020-06-17 09:52:30:566 INFO [ograph01-5-71]<-[ograph03-5-73] Received updated status ograph03-5-73.web=ONLINE [OHazelcastPlugin] 2020-06-17 09:52:33:214 INFO [ograph01-5-71] Distributed servers status (*=current @=lockmgr[ograph01-5-71]): +-------------------+------+-------------------+-----+---------+----------------+----------------+------------------------+ |Name |Status|Databases |Conns|StartedOn|Binary |HTTP |UsedMemory | +-------------------+------+-------------------+-----+---------+----------------+----------------+------------------------+ |ograph02-5-72 |ONLINE|web=ONLINE (MASTER)|4 |09:51:36 |172.16.5.72:2424|172.16.5.72:2480|738.93MB/64.00GB (1.13%)| |ograph01-5-71(*)(@)|ONLINE|web=ONLINE (MASTER)|4 |09:51:19 |172.16.5.71:2424|172.16.5.71:2480|757.35MB/64.00GB (1.16%)| |ograph03-5-73 |ONLINE|web=ONLINE (MASTER)|4 |09:52:09 |172.16.5.73:2424|172.16.5.73:2480|716.46MB/64.00GB (1.09%)| +-------------------+------+-------------------+-----+---------+----------------+----------------+------------------------+ [OHazelcastPlugin] 2020-06-17 09:52:51:409 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Accepting socket connection from /172.16.5.74:43153 [TcpIpAcceptor] 2020-06-17 09:52:51:409 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Established socket connection between /172.16.5.71:2434 and /172.16.5.74:43153 [TcpIpConnectionManager] 2020-06-17 09:52:58:417 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Members {size:4, ver:4} [ Member [172.16.5.71]:2434 - 111d390c-f6c1-48c3-890b-a004ae134975 this Member [172.16.5.72]:2434 - 6909ffd6-153d-49ce-9286-8defd6aff559 Member [172.16.5.73]:2434 - 6cbb8848-2d8f-44f9-aad9-556bae5e5c55 Member [172.16.5.74]:2434 - 69f3bf59-4710-4123-a88c-6bf7ef36349c ] [ClusterService] 2020-06-17 09:52:58:418 INFO [ograph01-5-71] Added new node id=Member [172.16.5.74]:2434 - 69f3bf59-4710-4123-a88c-6bf7ef36349c name=ext:69f3bf59-4710-4123-a88c-6bf7ef36349c [OHazelcastPlugin] 2020-06-17 09:52:58:671 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Re-partitioning cluster data... Migration queue size: 271 [MigrationManager] 2020-06-17 09:52:59:537 INFO [ograph01-5-71]<-[ext:69f3bf59-4710-4123-a88c-6bf7ef36349c] Received updated about registered nodes [OHazelcastPlugin] 2020-06-17 09:52:59:709 INFO [ograph01-5-71]<-[ograph04-5-74] Added node configuration id=Member [172.16.5.74]:2434 - 69f3bf59-4710-4123-a88c-6bf7ef36349c name=ograph04-5-74, now 4 nodes are configured [OHazelcastPlugin] 2020-06-17 09:52:59:712 INFO [ograph01-5-71] Distributed servers status (*=current @=lockmgr[ograph01-5-71]): +-------------------+--------+-------------------+-----+---------+----------------+----------------+------------------------+ |Name |Status |Databases |Conns|StartedOn|Binary |HTTP |UsedMemory | +-------------------+--------+-------------------+-----+---------+----------------+----------------+------------------------+ |ograph02-5-72 |ONLINE |web=ONLINE (MASTER)|4 |09:51:36 |172.16.5.72:2424|172.16.5.72:2480|752.32MB/64.00GB (1.15%)| |ograph01-5-71(*)(@)|ONLINE |web=ONLINE (MASTER)|4 |09:51:19 |172.16.5.71:2424|172.16.5.71:2480|768.79MB/64.00GB (1.17%)| |ograph04-5-74 |STARTING| |0 |09:52:49 |172.16.5.74:2424|172.16.5.74:2480|452.71MB/64.00GB (0.69%)| |ograph03-5-73 |ONLINE |web=ONLINE (MASTER)|4 |09:52:09 |172.16.5.73:2424|172.16.5.73:2480|731.62MB/64.00GB (1.12%)| +-------------------+--------+-------------------+-----+---------+----------------+----------------+------------------------+ [OHazelcastPlugin] 2020-06-17 09:52:59:713 INFO [ograph01-5-71] Distributed servers status (*=current @=lockmgr[ograph01-5-71]): +-------------------+------+-------------------+-----+---------+----------------+----------------+------------------------+ |Name |Status|Databases |Conns|StartedOn|Binary |HTTP |UsedMemory | +-------------------+------+-------------------+-----+---------+----------------+----------------+------------------------+ |ograph02-5-72 |ONLINE|web=ONLINE (MASTER)|4 |09:51:36 |172.16.5.72:2424|172.16.5.72:2480|752.32MB/64.00GB (1.15%)| |ograph01-5-71(*)(@)|ONLINE|web=ONLINE (MASTER)|4 |09:51:19 |172.16.5.71:2424|172.16.5.71:2480|768.79MB/64.00GB (1.17%)| |ograph04-5-74 | | | | | | | | |ograph03-5-73 |ONLINE|web=ONLINE (MASTER)|4 |09:52:09 |172.16.5.73:2424|172.16.5.73:2480|731.62MB/64.00GB (1.12%)| +-------------------+------+-------------------+-----+---------+----------------+----------------+------------------------+ [OHazelcastPlugin] 2020-06-17 09:52:59:733 INFO [ograph01-5-71] Distributed servers status (*=current @=lockmgr[ograph01-5-71]): +-------------------+--------+-------------------+-----+---------+----------------+----------------+------------------------+ |Name |Status |Databases |Conns|StartedOn|Binary |HTTP |UsedMemory | +-------------------+--------+-------------------+-----+---------+----------------+----------------+------------------------+ |ograph02-5-72 |ONLINE |web=ONLINE (MASTER)|4 |09:51:36 |172.16.5.72:2424|172.16.5.72:2480|752.32MB/64.00GB (1.15%)| |ograph01-5-71(*)(@)|ONLINE |web=ONLINE (MASTER)|4 |09:51:19 |172.16.5.71:2424|172.16.5.71:2480|768.79MB/64.00GB (1.17%)| |ograph04-5-74 |STARTING| |0 |09:52:49 |172.16.5.74:2424|172.16.5.74:2480|452.71MB/64.00GB (0.69%)| |ograph03-5-73 |ONLINE |web=ONLINE (MASTER)|4 |09:52:09 |172.16.5.73:2424|172.16.5.73:2480|731.62MB/64.00GB (1.12%)| +-------------------+--------+-------------------+-----+---------+----------------+----------------+------------------------+ [OHazelcastPlugin] 2020-06-17 09:52:59:762 INFO [ograph01-5-71]<-[ograph04-5-74] Received new status ograph04-5-74.OSystem=NOT_AVAILABLE [OHazelcastPlugin] 2020-06-17 09:53:01:222 INFO [ograph01-5-71] Received new status ograph04-5-74.web=SYNCHRONIZING [OHazelcastPlugin] 2020-06-17 09:53:01:222 INFO [ograph01-5-71]->[ograph04-5-74] Deploying database 'web' with delta of changes... [OSyncDatabaseDeltaTask] 2020-06-17 09:53:01:222 INFO [ograph01-5-71]->[ograph04-5-74] Creating delta backup of database 'web' (startLSN=OLogSequenceNumber{segment=14489, position=22540320}) in directory: /tmp/orientdb/backup_ograph04-5-74_web_server0.zip... [OSyncDatabaseDeltaTask] 2020-06-17 09:53:01:222 INFO Cannot find requested LSN=OLogSequenceNumber{segment=14489, position=22540320} for database sync operation (last available LSN is OLogSequenceNumber{segment=14491, position=8210}) [OLocalPaginatedStorage] 2020-06-17 09:53:01:235 INFO [ograph01-5-71]<-[ograph04-5-74] Received updated status ograph04-5-74.web=ONLINE [OHazelcastPlugin] 2020-06-17 09:53:01:239 INFO [ograph01-5-71]<-[ograph04-5-74] Received updated status ograph04-5-74.web=NOT_AVAILABLE [OHazelcastPlugin] 2020-06-17 09:53:01:570 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] All migration tasks have been completed, queues are empty. [MigrationThread] 2020-06-17 09:53:03:214 INFO [ograph01-5-71] Distributed servers status (*=current @=lockmgr[ograph01-5-71]): +-------------------+------+---------------------------+-----+---------+----------------+----------------+------------------------+ |Name |Status|Databases |Conns|StartedOn|Binary |HTTP |UsedMemory | +-------------------+------+---------------------------+-----+---------+----------------+----------------+------------------------+ |ograph02-5-72 |ONLINE|web=ONLINE (MASTER) |4 |09:51:36 |172.16.5.72:2424|172.16.5.72:2480|752.32MB/64.00GB (1.15%)| |ograph01-5-71(*)(@)|ONLINE|web=ONLINE (MASTER) |4 |09:51:19 |172.16.5.71:2424|172.16.5.71:2480|768.79MB/64.00GB (1.17%)| |ograph04-5-74 |ONLINE|web=NOT_AVAILABLE (REPLICA)|6 |09:52:49 |172.16.5.74:2424|172.16.5.74:2480|685.85MB/64.00GB (1.05%)| |ograph03-5-73 |ONLINE|web=ONLINE (MASTER) |6 |09:52:09 |172.16.5.73:2424|172.16.5.73:2480|758.69MB/64.00GB (1.16%)| +-------------------+------+---------------------------+-----+---------+----------------+----------------+------------------------+ [OHazelcastPlugin] 2020-06-17 09:53:11:339 INFO [ograph01-5-71] Received updated status ograph04-5-74.web=SYNCHRONIZING [OHazelcastPlugin] 2020-06-17 09:53:11:340 INFO [ograph01-5-71]->[ograph04-5-74] Deploying database 'web' with delta of changes... [OSyncDatabaseDeltaTask] 2020-06-17 09:53:11:341 INFO [ograph01-5-71]->[ograph04-5-74] Creating delta backup of database 'web' (startLSN=OLogSequenceNumber{segment=14489, position=22540320}) in directory: /tmp/orientdb/backup_ograph04-5-74_web_server0.zip... [OSyncDatabaseDeltaTask] 2020-06-17 09:53:11:341 INFO Cannot find requested LSN=OLogSequenceNumber{segment=14489, position=22540320} for database sync operation (last available LSN is OLogSequenceNumber{segment=14491, position=8210}) [OLocalPaginatedStorage] 2020-06-17 09:53:11:345 INFO [ograph01-5-71]<-[ograph04-5-74] Received updated status ograph04-5-74.web=ONLINE [OHazelcastPlugin] 2020-06-17 09:53:11:346 INFO [ograph01-5-71] Distributed servers status (*=current @=lockmgr[ograph01-5-71]): +-------------------+------+---------------------------+-----+---------+----------------+----------------+------------------------+ |Name |Status|Databases |Conns|StartedOn|Binary |HTTP |UsedMemory | +-------------------+------+---------------------------+-----+---------+----------------+----------------+------------------------+ |ograph02-5-72 |ONLINE|web=ONLINE (MASTER) |6 |09:51:36 |172.16.5.72:2424|172.16.5.72:2480|650.73MB/64.00GB (0.99%)| |ograph01-5-71(*)(@)|ONLINE|web=ONLINE (MASTER) |6 |09:51:19 |172.16.5.71:2424|172.16.5.71:2480|818.67MB/64.00GB (1.25%)| |ograph04-5-74 |ONLINE|web=SYNCHRONIZING (REPLICA)|6 |09:52:49 |172.16.5.74:2424|172.16.5.74:2480|692.11MB/64.00GB (1.06%)| |ograph03-5-73 |ONLINE|web=ONLINE (MASTER) |6 |09:52:09 |172.16.5.73:2424|172.16.5.73:2480|766.87MB/64.00GB (1.17%)| +-------------------+------+---------------------------+-----+---------+----------------+----------------+------------------------+ [OHazelcastPlugin] 2020-06-17 09:53:13:217 INFO [ograph01-5-71] Distributed servers status (*=current @=lockmgr[ograph01-5-71]): +-------------------+------+--------------------+-----+---------+----------------+----------------+------------------------+ |Name |Status|Databases |Conns|StartedOn|Binary |HTTP |UsedMemory | +-------------------+------+--------------------+-----+---------+----------------+----------------+------------------------+ |ograph02-5-72 |ONLINE|web=ONLINE (MASTER) |6 |09:51:36 |172.16.5.72:2424|172.16.5.72:2480|650.73MB/64.00GB (0.99%)| |ograph01-5-71(*)(@)|ONLINE|web=ONLINE (MASTER) |6 |09:51:19 |172.16.5.71:2424|172.16.5.71:2480|818.67MB/64.00GB (1.25%)| |ograph04-5-74 |ONLINE|web=ONLINE (REPLICA)|6 |09:52:49 |172.16.5.74:2424|172.16.5.74:2480|692.11MB/64.00GB (1.06%)| |ograph03-5-73 |ONLINE|web=ONLINE (MASTER) |6 |09:52:09 |172.16.5.73:2424|172.16.5.73:2480|766.87MB/64.00GB (1.17%)| +-------------------+------+--------------------+-----+---------+----------------+----------------+------------------------+ [OHazelcastPlugin] 2020-06-17 09:53:26:798 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Accepting socket connection from /172.16.5.75:55483 [TcpIpAcceptor] 2020-06-17 09:53:26:798 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Established socket connection between /172.16.5.71:2434 and /172.16.5.75:55483 [TcpIpConnectionManager] 2020-06-17 09:53:33:800 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Members {size:5, ver:5} [ Member [172.16.5.71]:2434 - 111d390c-f6c1-48c3-890b-a004ae134975 this Member [172.16.5.72]:2434 - 6909ffd6-153d-49ce-9286-8defd6aff559 Member [172.16.5.73]:2434 - 6cbb8848-2d8f-44f9-aad9-556bae5e5c55 Member [172.16.5.74]:2434 - 69f3bf59-4710-4123-a88c-6bf7ef36349c Member [172.16.5.75]:2434 - 3ce6c7e3-9978-4357-b73d-0040424af433 ] [ClusterService] 2020-06-17 09:53:33:803 INFO [ograph01-5-71] Added new node id=Member [172.16.5.75]:2434 - 3ce6c7e3-9978-4357-b73d-0040424af433 name=ext:3ce6c7e3-9978-4357-b73d-0040424af433 [OHazelcastPlugin] 2020-06-17 09:53:34:056 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Re-partitioning cluster data... Migration queue size: 350 [MigrationManager] 2020-06-17 09:53:34:953 INFO [ograph01-5-71]<-[ext:3ce6c7e3-9978-4357-b73d-0040424af433] Received updated about registered nodes [OHazelcastPlugin] 2020-06-17 09:53:35:429 INFO [ograph01-5-71]<-[ograph05-5-75] Added node configuration id=Member [172.16.5.75]:2434 - 3ce6c7e3-9978-4357-b73d-0040424af433 name=ograph05-5-75, now 5 nodes are configured [OHazelcastPlugin] 2020-06-17 09:53:35:434 INFO [ograph01-5-71] Distributed servers status (*=current @=lockmgr[ograph01-5-71]): +-------------------+--------+--------------------+-----+---------+----------------+----------------+------------------------+ |Name |Status |Databases |Conns|StartedOn|Binary |HTTP |UsedMemory | +-------------------+--------+--------------------+-----+---------+----------------+----------------+------------------------+ |ograph02-5-72 |ONLINE |web=ONLINE (MASTER) |6 |09:51:36 |172.16.5.72:2424|172.16.5.72:2480|658.24MB/64.00GB (1.00%)| |ograph05-5-75 |STARTING| |0 |09:53:25 |172.16.5.75:2424|172.16.5.75:2480|464.77MB/64.00GB (0.71%)| |ograph01-5-71(*)(@)|ONLINE |web=ONLINE (MASTER) |6 |09:51:19 |172.16.5.71:2424|172.16.5.71:2480|830.87MB/64.00GB (1.27%)| |ograph04-5-74 |ONLINE |web=ONLINE (REPLICA)|6 |09:52:49 |172.16.5.74:2424|172.16.5.74:2480|709.55MB/64.00GB (1.08%)| |ograph03-5-73 |ONLINE |web=ONLINE (MASTER) |6 |09:52:09 |172.16.5.73:2424|172.16.5.73:2480|776.27MB/64.00GB (1.18%)| +-------------------+--------+--------------------+-----+---------+----------------+----------------+------------------------+ [OHazelcastPlugin] 2020-06-17 09:53:35:436 INFO [ograph01-5-71] Distributed servers status (*=current @=lockmgr[ograph01-5-71]): +-------------------+------+--------------------+-----+---------+----------------+----------------+------------------------+ |Name |Status|Databases |Conns|StartedOn|Binary |HTTP |UsedMemory | +-------------------+------+--------------------+-----+---------+----------------+----------------+------------------------+ |ograph02-5-72 |ONLINE|web=ONLINE (MASTER) |6 |09:51:36 |172.16.5.72:2424|172.16.5.72:2480|658.24MB/64.00GB (1.00%)| |ograph05-5-75 | | | | | | | | |ograph01-5-71(*)(@)|ONLINE|web=ONLINE (MASTER) |6 |09:51:19 |172.16.5.71:2424|172.16.5.71:2480|830.87MB/64.00GB (1.27%)| |ograph04-5-74 |ONLINE|web=ONLINE (REPLICA)|6 |09:52:49 |172.16.5.74:2424|172.16.5.74:2480|709.55MB/64.00GB (1.08%)| |ograph03-5-73 |ONLINE|web=ONLINE (MASTER) |6 |09:52:09 |172.16.5.73:2424|172.16.5.73:2480|776.27MB/64.00GB (1.18%)| +-------------------+------+--------------------+-----+---------+----------------+----------------+------------------------+ [OHazelcastPlugin] 2020-06-17 09:53:35:438 INFO [ograph01-5-71] Distributed servers status (*=current @=lockmgr[ograph01-5-71]): +-------------------+--------+--------------------+-----+---------+----------------+----------------+------------------------+ |Name |Status |Databases |Conns|StartedOn|Binary |HTTP |UsedMemory | +-------------------+--------+--------------------+-----+---------+----------------+----------------+------------------------+ |ograph02-5-72 |ONLINE |web=ONLINE (MASTER) |6 |09:51:36 |172.16.5.72:2424|172.16.5.72:2480|658.24MB/64.00GB (1.00%)| |ograph05-5-75 |STARTING| |0 |09:53:25 |172.16.5.75:2424|172.16.5.75:2480|463.05MB/64.00GB (0.71%)| |ograph01-5-71(*)(@)|ONLINE |web=ONLINE (MASTER) |6 |09:51:19 |172.16.5.71:2424|172.16.5.71:2480|830.87MB/64.00GB (1.27%)| |ograph04-5-74 |ONLINE |web=ONLINE (REPLICA)|6 |09:52:49 |172.16.5.74:2424|172.16.5.74:2480|709.55MB/64.00GB (1.08%)| |ograph03-5-73 |ONLINE |web=ONLINE (MASTER) |6 |09:52:09 |172.16.5.73:2424|172.16.5.73:2480|776.27MB/64.00GB (1.18%)| +-------------------+--------+--------------------+-----+---------+----------------+----------------+------------------------+ [OHazelcastPlugin] 2020-06-17 09:53:35:438 INFO [ograph01-5-71]<-[ograph05-5-75] Received new status ograph05-5-75.OSystem=NOT_AVAILABLE [OHazelcastPlugin] 2020-06-17 09:53:35:746 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Remaining migration tasks in queue => 153 [InternalPartitionService] 2020-06-17 09:53:37:070 INFO [ograph01-5-71]<-[ograph02-5-72] Received new status ograph05-5-75.web=SYNCHRONIZING [OHazelcastPlugin] 2020-06-17 09:53:37:090 INFO [ograph01-5-71]<-[ograph05-5-75] Received updated status ograph05-5-75.web=ONLINE [OHazelcastPlugin] 2020-06-17 09:53:37:095 INFO [ograph01-5-71]<-[ograph05-5-75] Received updated status ograph05-5-75.web=NOT_AVAILABLE [OHazelcastPlugin] 2020-06-17 09:53:37:604 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] All migration tasks have been completed, queues are empty. [MigrationThread] 2020-06-17 09:53:40:457 INFO [ograph01-5-71] Distributed servers status (*=current @=lockmgr[ograph01-5-71]): +-------------------+------+---------------------------+-----+---------+----------------+----------------+------------------------+ |Name |Status|Databases |Conns|StartedOn|Binary |HTTP |UsedMemory | +-------------------+------+---------------------------+-----+---------+----------------+----------------+------------------------+ |ograph02-5-72 |ONLINE|web=ONLINE (MASTER) |8 |09:51:36 |172.16.5.72:2424|172.16.5.72:2480|695.21MB/64.00GB (1.06%)| |ograph05-5-75 |ONLINE|web=NOT_AVAILABLE (REPLICA)|8 |09:53:25 |172.16.5.75:2424|172.16.5.75:2480|711.94MB/64.00GB (1.09%)| |ograph01-5-71(*)(@)|ONLINE|web=ONLINE (MASTER) |6 |09:51:19 |172.16.5.71:2424|172.16.5.71:2480|830.87MB/64.00GB (1.27%)| |ograph04-5-74 |ONLINE|web=ONLINE (REPLICA) |6 |09:52:49 |172.16.5.74:2424|172.16.5.74:2480|709.55MB/64.00GB (1.08%)| |ograph03-5-73 |ONLINE|web=ONLINE (MASTER) |6 |09:52:09 |172.16.5.73:2424|172.16.5.73:2480|776.27MB/64.00GB (1.18%)| +-------------------+------+---------------------------+-----+---------+----------------+----------------+------------------------+ [OHazelcastPlugin] 2020-06-17 09:53:47:209 INFO [ograph01-5-71]<-[ograph02-5-72] Received updated status ograph05-5-75.web=SYNCHRONIZING [OHazelcastPlugin] 2020-06-17 09:53:47:223 INFO [ograph01-5-71]<-[ograph05-5-75] Received updated status ograph05-5-75.web=ONLINE [OHazelcastPlugin] 2020-06-17 09:53:47:224 INFO [ograph01-5-71] Distributed servers status (*=current @=lockmgr[ograph01-5-71]): +-------------------+------+---------------------------+-----+---------+----------------+----------------+------------------------+ |Name |Status|Databases |Conns|StartedOn|Binary |HTTP |UsedMemory | +-------------------+------+---------------------------+-----+---------+----------------+----------------+------------------------+ |ograph02-5-72 |ONLINE|web=ONLINE (MASTER) |8 |09:51:36 |172.16.5.72:2424|172.16.5.72:2480|699.79MB/64.00GB (1.07%)| |ograph05-5-75 |ONLINE|web=SYNCHRONIZING (REPLICA)|8 |09:53:25 |172.16.5.75:2424|172.16.5.75:2480|716.58MB/64.00GB (1.09%)| |ograph01-5-71(*)(@)|ONLINE|web=ONLINE (MASTER) |8 |09:51:19 |172.16.5.71:2424|172.16.5.71:2480|761.08MB/64.00GB (1.16%)| |ograph04-5-74 |ONLINE|web=ONLINE (REPLICA) |8 |09:52:49 |172.16.5.74:2424|172.16.5.74:2480|747.73MB/64.00GB (1.14%)| |ograph03-5-73 |ONLINE|web=ONLINE (MASTER) |8 |09:52:09 |172.16.5.73:2424|172.16.5.73:2480|685.07MB/64.00GB (1.05%)| +-------------------+------+---------------------------+-----+---------+----------------+----------------+------------------------+ [OHazelcastPlugin] 2020-06-17 09:53:50:456 INFO [ograph01-5-71] Distributed servers status (*=current @=lockmgr[ograph01-5-71]): +-------------------+------+--------------------+-----+---------+----------------+----------------+------------------------+ |Name |Status|Databases |Conns|StartedOn|Binary |HTTP |UsedMemory | +-------------------+------+--------------------+-----+---------+----------------+----------------+------------------------+ |ograph02-5-72 |ONLINE|web=ONLINE (MASTER) |8 |09:51:36 |172.16.5.72:2424|172.16.5.72:2480|699.79MB/64.00GB (1.07%)| |ograph05-5-75 |ONLINE|web=ONLINE (REPLICA)|8 |09:53:25 |172.16.5.75:2424|172.16.5.75:2480|716.58MB/64.00GB (1.09%)| |ograph01-5-71(*)(@)|ONLINE|web=ONLINE (MASTER) |8 |09:51:19 |172.16.5.71:2424|172.16.5.71:2480|761.08MB/64.00GB (1.16%)| |ograph04-5-74 |ONLINE|web=ONLINE (REPLICA)|8 |09:52:49 |172.16.5.74:2424|172.16.5.74:2480|747.73MB/64.00GB (1.14%)| |ograph03-5-73 |ONLINE|web=ONLINE (MASTER) |8 |09:52:09 |172.16.5.73:2424|172.16.5.73:2480|685.07MB/64.00GB (1.05%)| +-------------------+------+--------------------+-----+---------+----------------+----------------+------------------------+ [OHazelcastPlugin] 2020-06-17 09:54:12:033 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Accepting socket connection from /172.16.5.76:41063 [TcpIpAcceptor] 2020-06-17 09:54:12:033 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Established socket connection between /172.16.5.71:2434 and /172.16.5.76:41063 [TcpIpConnectionManager] 2020-06-17 09:54:19:047 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Members {size:6, ver:6} [ Member [172.16.5.71]:2434 - 111d390c-f6c1-48c3-890b-a004ae134975 this Member [172.16.5.72]:2434 - 6909ffd6-153d-49ce-9286-8defd6aff559 Member [172.16.5.73]:2434 - 6cbb8848-2d8f-44f9-aad9-556bae5e5c55 Member [172.16.5.74]:2434 - 69f3bf59-4710-4123-a88c-6bf7ef36349c Member [172.16.5.75]:2434 - 3ce6c7e3-9978-4357-b73d-0040424af433 Member [172.16.5.76]:2434 - 67f4dfe5-ab69-4b5e-9435-33114322a2be ] [ClusterService] 2020-06-17 09:54:19:057 INFO [ograph01-5-71] Added new node id=Member [172.16.5.76]:2434 - 67f4dfe5-ab69-4b5e-9435-33114322a2be name=ext:67f4dfe5-ab69-4b5e-9435-33114322a2be [OHazelcastPlugin] 2020-06-17 09:54:19:122 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Re-partitioning cluster data... Migration queue size: 468 [MigrationManager] 2020-06-17 09:54:20:167 INFO [ograph01-5-71]<-[ext:67f4dfe5-ab69-4b5e-9435-33114322a2be] Received updated about registered nodes [OHazelcastPlugin] 2020-06-17 09:54:20:410 INFO [ograph01-5-71]<-[ograph06-5-76] Added node configuration id=Member [172.16.5.76]:2434 - 67f4dfe5-ab69-4b5e-9435-33114322a2be name=ograph06-5-76, now 6 nodes are configured [OHazelcastPlugin] 2020-06-17 09:54:20:419 INFO [ograph01-5-71] Distributed servers status (*=current @=lockmgr[ograph01-5-71]): +-------------------+--------+--------------------+-----+---------+----------------+----------------+------------------------+ |Name |Status |Databases |Conns|StartedOn|Binary |HTTP |UsedMemory | +-------------------+--------+--------------------+-----+---------+----------------+----------------+------------------------+ |ograph02-5-72 |ONLINE |web=ONLINE (MASTER) |8 |09:51:36 |172.16.5.72:2424|172.16.5.72:2480|713.10MB/64.00GB (1.09%)| |ograph05-5-75 |ONLINE |web=ONLINE (REPLICA)|8 |09:53:25 |172.16.5.75:2424|172.16.5.75:2480|735.09MB/64.00GB (1.12%)| |ograph01-5-71(*)(@)|ONLINE |web=ONLINE (MASTER) |8 |09:51:19 |172.16.5.71:2424|172.16.5.71:2480|772.81MB/64.00GB (1.18%)| |ograph04-5-74 |ONLINE |web=ONLINE (REPLICA)|8 |09:52:49 |172.16.5.74:2424|172.16.5.74:2480|763.69MB/64.00GB (1.17%)| |ograph03-5-73 |ONLINE |web=ONLINE (MASTER) |8 |09:52:09 |172.16.5.73:2424|172.16.5.73:2480|697.18MB/64.00GB (1.06%)| |ograph06-5-76 |STARTING| |2 |09:54:10 |172.16.5.76:2424|172.16.5.76:2480|464.72MB/64.00GB (0.71%)| +-------------------+--------+--------------------+-----+---------+----------------+----------------+------------------------+ [OHazelcastPlugin] 2020-06-17 09:54:20:420 INFO [ograph01-5-71] Distributed servers status (*=current @=lockmgr[ograph01-5-71]): +-------------------+------+--------------------+-----+---------+----------------+----------------+------------------------+ |Name |Status|Databases |Conns|StartedOn|Binary |HTTP |UsedMemory | +-------------------+------+--------------------+-----+---------+----------------+----------------+------------------------+ |ograph02-5-72 |ONLINE|web=ONLINE (MASTER) |8 |09:51:36 |172.16.5.72:2424|172.16.5.72:2480|713.10MB/64.00GB (1.09%)| |ograph05-5-75 |ONLINE|web=ONLINE (REPLICA)|8 |09:53:25 |172.16.5.75:2424|172.16.5.75:2480|735.09MB/64.00GB (1.12%)| |ograph01-5-71(*)(@)|ONLINE|web=ONLINE (MASTER) |8 |09:51:19 |172.16.5.71:2424|172.16.5.71:2480|772.81MB/64.00GB (1.18%)| |ograph04-5-74 |ONLINE|web=ONLINE (REPLICA)|8 |09:52:49 |172.16.5.74:2424|172.16.5.74:2480|763.69MB/64.00GB (1.17%)| |ograph03-5-73 |ONLINE|web=ONLINE (MASTER) |8 |09:52:09 |172.16.5.73:2424|172.16.5.73:2480|697.18MB/64.00GB (1.06%)| |ograph06-5-76 | | | | | | | | +-------------------+------+--------------------+-----+---------+----------------+----------------+------------------------+ [OHazelcastPlugin] 2020-06-17 09:54:20:421 INFO [ograph01-5-71] Distributed servers status (*=current @=lockmgr[ograph01-5-71]): +-------------------+--------+--------------------+-----+---------+----------------+----------------+------------------------+ |Name |Status |Databases |Conns|StartedOn|Binary |HTTP |UsedMemory | +-------------------+--------+--------------------+-----+---------+----------------+----------------+------------------------+ |ograph02-5-72 |ONLINE |web=ONLINE (MASTER) |8 |09:51:36 |172.16.5.72:2424|172.16.5.72:2480|713.10MB/64.00GB (1.09%)| |ograph05-5-75 |ONLINE |web=ONLINE (REPLICA)|8 |09:53:25 |172.16.5.75:2424|172.16.5.75:2480|735.09MB/64.00GB (1.12%)| |ograph01-5-71(*)(@)|ONLINE |web=ONLINE (MASTER) |8 |09:51:19 |172.16.5.71:2424|172.16.5.71:2480|772.81MB/64.00GB (1.18%)| |ograph04-5-74 |ONLINE |web=ONLINE (REPLICA)|8 |09:52:49 |172.16.5.74:2424|172.16.5.74:2480|763.69MB/64.00GB (1.17%)| |ograph03-5-73 |ONLINE |web=ONLINE (MASTER) |8 |09:52:09 |172.16.5.73:2424|172.16.5.73:2480|697.18MB/64.00GB (1.06%)| |ograph06-5-76 |STARTING| |0 |09:54:10 |172.16.5.76:2424|172.16.5.76:2480|462.35MB/64.00GB (0.71%)| +-------------------+--------+--------------------+-----+---------+----------------+----------------+------------------------+ [OHazelcastPlugin] 2020-06-17 09:54:20:422 INFO [ograph01-5-71]<-[ograph06-5-76] Received new status ograph06-5-76.OSystem=NOT_AVAILABLE [OHazelcastPlugin] 2020-06-17 09:54:20:746 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Remaining migration tasks in queue => 209 [InternalPartitionService] 2020-06-17 09:54:22:639 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] All migration tasks have been completed, queues are empty. [MigrationThread] 2020-06-17 09:54:27:468 INFO [ograph01-5-71]<-[ograph02-5-72] Received new status ograph06-5-76.web=SYNCHRONIZING [OHazelcastPlugin] 2020-06-17 09:54:27:482 INFO [ograph01-5-71]<-[ograph06-5-76] Received updated status ograph06-5-76.web=ONLINE [OHazelcastPlugin] 2020-06-17 09:54:27:485 INFO [ograph01-5-71]<-[ograph06-5-76] Received updated status ograph06-5-76.web=NOT_AVAILABLE [OHazelcastPlugin] 2020-06-17 09:54:27:526 INFO [ograph01-5-71] Distributed servers status (*=current @=lockmgr[ograph01-5-71]): +-------------------+------+---------------------------+-----+---------+----------------+----------------+------------------------+ |Name |Status|Databases |Conns|StartedOn|Binary |HTTP |UsedMemory | +-------------------+------+---------------------------+-----+---------+----------------+----------------+------------------------+ |ograph02-5-72 |ONLINE|web=ONLINE (MASTER) |10 |09:51:36 |172.16.5.72:2424|172.16.5.72:2480|758.31MB/64.00GB (1.16%)| |ograph05-5-75 |ONLINE|web=ONLINE (REPLICA) |10 |09:53:25 |172.16.5.75:2424|172.16.5.75:2480|659.83MB/64.00GB (1.01%)| |ograph01-5-71(*)(@)|ONLINE|web=ONLINE (MASTER) |10 |09:51:19 |172.16.5.71:2424|172.16.5.71:2480|857.82MB/64.00GB (1.31%)| |ograph04-5-74 |ONLINE|web=ONLINE (REPLICA) |10 |09:52:49 |172.16.5.74:2424|172.16.5.74:2480|666.76MB/64.00GB (1.02%)| |ograph03-5-73 |ONLINE|web=ONLINE (MASTER) |10 |09:52:09 |172.16.5.73:2424|172.16.5.73:2480|720.02MB/64.00GB (1.10%)| |ograph06-5-76 |ONLINE|web=NOT_AVAILABLE (REPLICA)|10 |09:54:10 |172.16.5.76:2424|172.16.5.76:2480|1.22GB/64.00GB (1.90%) | +-------------------+------+---------------------------+-----+---------+----------------+----------------+------------------------+ [OHazelcastPlugin] 2020-06-17 09:54:37:548 INFO [ograph01-5-71]<-[ograph02-5-72] Received updated status ograph06-5-76.web=SYNCHRONIZING [OHazelcastPlugin] 2020-06-17 09:54:37:550 INFO [ograph01-5-71] Distributed servers status (*=current @=lockmgr[ograph01-5-71]): +-------------------+------+---------------------------+-----+---------+----------------+----------------+------------------------+ |Name |Status|Databases |Conns|StartedOn|Binary |HTTP |UsedMemory | +-------------------+------+---------------------------+-----+---------+----------------+----------------+------------------------+ |ograph02-5-72 |ONLINE|web=ONLINE (MASTER) |10 |09:51:36 |172.16.5.72:2424|172.16.5.72:2480|762.51MB/64.00GB (1.16%)| |ograph05-5-75 |ONLINE|web=ONLINE (REPLICA) |10 |09:53:25 |172.16.5.75:2424|172.16.5.75:2480|664.21MB/64.00GB (1.01%)| |ograph01-5-71(*)(@)|ONLINE|web=ONLINE (MASTER) |10 |09:51:19 |172.16.5.71:2424|172.16.5.71:2480|861.46MB/64.00GB (1.31%)| |ograph04-5-74 |ONLINE|web=ONLINE (REPLICA) |10 |09:52:49 |172.16.5.74:2424|172.16.5.74:2480|681.70MB/64.00GB (1.04%)| |ograph03-5-73 |ONLINE|web=ONLINE (MASTER) |10 |09:52:09 |172.16.5.73:2424|172.16.5.73:2480|748.11MB/64.00GB (1.14%)| |ograph06-5-76 |ONLINE|web=SYNCHRONIZING (REPLICA)|10 |09:54:10 |172.16.5.76:2424|172.16.5.76:2480|1.23GB/64.00GB (1.92%) | +-------------------+------+---------------------------+-----+---------+----------------+----------------+------------------------+ [OHazelcastPlugin] 2020-06-17 09:54:37:554 INFO [ograph01-5-71]<-[ograph06-5-76] Received updated status ograph06-5-76.web=ONLINE [OHazelcastPlugin] 2020-06-17 09:54:37:555 INFO [ograph01-5-71] Distributed servers status (*=current @=lockmgr[ograph01-5-71]): +-------------------+------+--------------------+-----+---------+----------------+----------------+------------------------+ |Name |Status|Databases |Conns|StartedOn|Binary |HTTP |UsedMemory | +-------------------+------+--------------------+-----+---------+----------------+----------------+------------------------+ |ograph02-5-72 |ONLINE|web=ONLINE (MASTER) |10 |09:51:36 |172.16.5.72:2424|172.16.5.72:2480|762.51MB/64.00GB (1.16%)| |ograph05-5-75 |ONLINE|web=ONLINE (REPLICA)|10 |09:53:25 |172.16.5.75:2424|172.16.5.75:2480|664.21MB/64.00GB (1.01%)| |ograph01-5-71(*)(@)|ONLINE|web=ONLINE (MASTER) |10 |09:51:19 |172.16.5.71:2424|172.16.5.71:2480|861.46MB/64.00GB (1.31%)| |ograph04-5-74 |ONLINE|web=ONLINE (REPLICA)|10 |09:52:49 |172.16.5.74:2424|172.16.5.74:2480|681.70MB/64.00GB (1.04%)| |ograph03-5-73 |ONLINE|web=ONLINE (MASTER) |10 |09:52:09 |172.16.5.73:2424|172.16.5.73:2480|748.11MB/64.00GB (1.14%)| |ograph06-5-76 |ONLINE|web=ONLINE (REPLICA)|10 |09:54:10 |172.16.5.76:2424|172.16.5.76:2480|1.23GB/64.00GB (1.92%) | +-------------------+------+--------------------+-----+---------+----------------+----------------+------------------------+ [OHazelcastPlugin] 2020-06-17 09:55:21:945 INFO [ograph01-5-71]<-[ograph05-5-75] Server ograph05-5-75 is waiting to acquire distributed lock on resource 'web' owned by ograph06-5-76 on Wed Jun 17 09:55:21 ICT 2020 (threadId=76 timeout=20000)... [ODistributedLockManagerExecutor] 2020-06-17 09:55:22:004 INFO [ograph01-5-71]<-[ograph02-5-72] Received updated status ograph06-5-76.web=SYNCHRONIZING [OHazelcastPlugin] 2020-06-17 09:55:22:006 INFO [ograph01-5-71] Distributed servers status (*=current @=lockmgr[ograph01-5-71]): +-------------------+------+---------------------------+-----+---------+----------------+----------------+------------------------+ |Name |Status|Databases |Conns|StartedOn|Binary |HTTP |UsedMemory | +-------------------+------+---------------------------+-----+---------+----------------+----------------+------------------------+ |ograph02-5-72 |ONLINE|web=ONLINE (MASTER) |11 |09:51:36 |172.16.5.72:2424|172.16.5.72:2480|790.03MB/64.00GB (1.21%)| |ograph05-5-75 |ONLINE|web=ONLINE (REPLICA) |10 |09:53:25 |172.16.5.75:2424|172.16.5.75:2480|684.39MB/64.00GB (1.04%)| |ograph01-5-71(*)(@)|ONLINE|web=ONLINE (MASTER) |26 |09:51:19 |172.16.5.71:2424|172.16.5.71:2480|1.25GB/64.00GB (1.95%) | |ograph04-5-74 |ONLINE|web=ONLINE (REPLICA) |10 |09:52:49 |172.16.5.74:2424|172.16.5.74:2480|752.59MB/64.00GB (1.15%)| |ograph03-5-73 |ONLINE|web=ONLINE (MASTER) |11 |09:52:09 |172.16.5.73:2424|172.16.5.73:2480|775.77MB/64.00GB (1.18%)| |ograph06-5-76 |ONLINE|web=SYNCHRONIZING (REPLICA)|10 |09:54:10 |172.16.5.76:2424|172.16.5.76:2480|1.28GB/64.00GB (1.99%) | +-------------------+------+---------------------------+-----+---------+----------------+----------------+------------------------+ [OHazelcastPlugin] 2020-06-17 09:55:43:225 WARNI [ograph01-5-71] Timeout (20001ms) on waiting for synchronous responses from nodes=[ograph02-5-72, ograph05-5-75, ograph04-5-74, ograph03-5-73, ograph06-5-76] responsesSoFar=[ograph02-5-72, ograph05-5-75, ograph04-5-74, ograph03-5-73] request=(id=0.654 task=gossip timestamp: 1592362523222 lockManagerServer: ograph01-5-71) [ODistributedDatabaseImpl] 2020-06-17 09:56:33:809 INFO [ograph01-5-71]<-[ograph06-5-76] Received updated status ograph06-5-76.web=ONLINE [OHazelcastPlugin] 2020-06-17 09:56:33:820 INFO [ograph01-5-71] Distributed servers status (*=current @=lockmgr[ograph01-5-71]): +-------------------+------+--------------------+-----+---------+----------------+----------------+------------------------+ |Name |Status|Databases |Conns|StartedOn|Binary |HTTP |UsedMemory | +-------------------+------+--------------------+-----+---------+----------------+----------------+------------------------+ |ograph02-5-72 |ONLINE|web=ONLINE (MASTER) |28 |09:51:36 |172.16.5.72:2424|172.16.5.72:2480|889.90MB/64.00GB (1.36%)| |ograph05-5-75 |ONLINE|web=ONLINE (REPLICA)|12 |09:53:25 |172.16.5.75:2424|172.16.5.75:2480|714.83MB/64.00GB (1.09%)| |ograph01-5-71(*)(@)|ONLINE|web=ONLINE (MASTER) |29 |09:51:19 |172.16.5.71:2424|172.16.5.71:2480|2.05GB/64.00GB (3.21%) | |ograph04-5-74 |ONLINE|web=ONLINE (REPLICA)|10 |09:52:49 |172.16.5.74:2424|172.16.5.74:2480|719.85MB/64.00GB (1.10%)| |ograph03-5-73 |ONLINE|web=ONLINE (MASTER) |11 |09:52:09 |172.16.5.73:2424|172.16.5.73:2480|943.21MB/64.00GB (1.44%)| |ograph06-5-76 |ONLINE|web=ONLINE (REPLICA)|13 |09:54:10 |172.16.5.76:2424|172.16.5.76:2480|1.72GB/64.00GB (2.68%) | +-------------------+------+--------------------+-----+---------+----------------+----------------+------------------------+ [OHazelcastPlugin] 2020-06-17 09:56:39:516 INFO [ograph01-5-71]<-[ograph05-5-75] Server ograph05-5-75 is waiting to acquire distributed lock on resource 'web' owned by ograph06-5-76 on Wed Jun 17 09:55:21 ICT 2020 (threadId=76 timeout=20000)... [ODistributedLockManagerExecutor] 2020-06-17 09:57:52:734 WARNI Received signal: SIGTERM [OSignalHandler] 2020-06-17 09:57:52:734 INFO OrientDB Server is shutting down... [OServer] 2020-06-17 09:57:52:735 INFO Shutting down listeners: [OServer] 2020-06-17 09:57:52:735 INFO - ONetworkProtocolBinary /0.0.0.0:2424: [OServer] 2020-06-17 09:57:52:735 INFO - ONetworkProtocolHttpDb /0.0.0.0:2480: [OServer] 2020-06-17 09:57:52:735 INFO Shutting down protocols [OServer] 2020-06-17 09:57:54:172 INFO [ograph01-5-71]<-[ograph02-5-72] Received updated status ograph01-5-71.web=NOT_AVAILABLE [OHazelcastPlugin] 2020-06-17 09:57:54:173 INFO [ograph01-5-71] Distributed servers status (*=current @=lockmgr[ograph02-5-72]): +----------------+------+--------------------------+-----+---------+----------------+----------------+------------------------+ |Name |Status|Databases |Conns|StartedOn|Binary |HTTP |UsedMemory | +----------------+------+--------------------------+-----+---------+----------------+----------------+------------------------+ |ograph02-5-72(@)|ONLINE|web=ONLINE (MASTER) |10 |09:51:36 |172.16.5.72:2424|172.16.5.72:2480|1.04GB/64.00GB (1.63%) | |ograph05-5-75 |ONLINE|web=ONLINE (REPLICA) |10 |09:53:25 |172.16.5.75:2424|172.16.5.75:2480|2.19GB/64.00GB (3.43%) | |ograph01-5-71(*)|ONLINE|web=NOT_AVAILABLE (MASTER)|29 |09:51:19 |172.16.5.71:2424|172.16.5.71:2480|2.60GB/64.00GB (4.06%) | |ograph04-5-74 |ONLINE|web=ONLINE (REPLICA) |10 |09:52:49 |172.16.5.74:2424|172.16.5.74:2480|827.97MB/64.00GB (1.26%)| |ograph03-5-73 |ONLINE|web=ONLINE (MASTER) |10 |09:52:09 |172.16.5.73:2424|172.16.5.73:2480|786.24MB/64.00GB (1.20%)| |ograph06-5-76 |ONLINE|web=ONLINE (REPLICA) |12 |09:54:10 |172.16.5.76:2424|172.16.5.76:2480|1.84GB/64.00GB (2.88%) | +----------------+------+--------------------------+-----+---------+----------------+----------------+------------------------+ [OHazelcastPlugin] 2020-06-17 09:57:54:173 INFO [ograph01-5-71]<-[ograph02-5-72] Received updated status ograph01-5-71.web=NOT_AVAILABLE [OHazelcastPlugin] 2020-06-17 09:57:54:174 INFO [ograph01-5-71]<-[ograph02-5-72] Received updated status ograph01-5-71.web=NOT_AVAILABLE [OHazelcastPlugin] 2020-06-17 09:57:54:174 INFO [ograph01-5-71]<-[ograph02-5-72] Received updated status ograph01-5-71.web=NOT_AVAILABLE [OHazelcastPlugin] 2020-06-17 09:57:54:175 INFO [ograph01-5-71]<-[ograph02-5-72] Received updated status ograph01-5-71.web=NOT_AVAILABLE [OHazelcastPlugin] 2020-06-17 09:57:58:138 INFO [ograph01-5-71]<-[ograph02-5-72] Received updated status ograph02-5-72.web=NOT_AVAILABLE [OHazelcastPlugin] 2020-06-17 09:57:58:139 INFO [ograph01-5-71] Distributed servers status (*=current @=lockmgr[ograph02-5-72]): +----------------+------+--------------------------+-----+---------+----------------+----------------+------------------------+ |Name |Status|Databases |Conns|StartedOn|Binary |HTTP |UsedMemory | +----------------+------+--------------------------+-----+---------+----------------+----------------+------------------------+ |ograph02-5-72(@)|ONLINE|web=NOT_AVAILABLE (MASTER)|10 |09:51:36 |172.16.5.72:2424|172.16.5.72:2480|1.91GB/64.00GB (2.98%) | |ograph05-5-75 |ONLINE|web=ONLINE (REPLICA) |10 |09:53:25 |172.16.5.75:2424|172.16.5.75:2480|2.19GB/64.00GB (3.43%) | |ograph01-5-71(*)|ONLINE|web=NOT_AVAILABLE (MASTER)|29 |09:51:19 |172.16.5.71:2424|172.16.5.71:2480|2.60GB/64.00GB (4.06%) | |ograph04-5-74 |ONLINE|web=ONLINE (REPLICA) |10 |09:52:49 |172.16.5.74:2424|172.16.5.74:2480|827.97MB/64.00GB (1.26%)| |ograph03-5-73 |ONLINE|web=ONLINE (MASTER) |10 |09:52:09 |172.16.5.73:2424|172.16.5.73:2480|786.24MB/64.00GB (1.20%)| |ograph06-5-76 |ONLINE|web=ONLINE (REPLICA) |12 |09:54:10 |172.16.5.76:2424|172.16.5.76:2480|1.84GB/64.00GB (2.88%) | +----------------+------+--------------------------+-----+---------+----------------+----------------+------------------------+ [OHazelcastPlugin] 2020-06-17 09:57:58:937 WARNI [ograph01-5-71] Timeout (80002ms) on waiting for synchronous responses from nodes=[ograph02-5-72, ograph05-5-75, ograph04-5-74, ograph03-5-73, ograph06-5-76] responsesSoFar=[ograph05-5-75, ograph04-5-74, ograph03-5-73, ograph06-5-76] request=(id=0.2130 task=upd_db_status) [ODistributedDatabaseImpl] 2020-06-17 09:58:03:674 INFO [ograph01-5-71]<-[ograph03-5-73] Received updated status ograph03-5-73.web=NOT_AVAILABLE [OHazelcastPlugin] 2020-06-17 09:58:03:675 INFO [ograph01-5-71] Distributed servers status (*=current @=lockmgr[ograph02-5-72]): +----------------+------+--------------------------+-----+---------+----------------+----------------+------------------------+ |Name |Status|Databases |Conns|StartedOn|Binary |HTTP |UsedMemory | +----------------+------+--------------------------+-----+---------+----------------+----------------+------------------------+ |ograph02-5-72(@)|ONLINE|web=NOT_AVAILABLE (MASTER)|10 |09:51:36 |172.16.5.72:2424|172.16.5.72:2480|1.91GB/64.00GB (2.98%) | |ograph05-5-75 |ONLINE|web=ONLINE (REPLICA) |10 |09:53:25 |172.16.5.75:2424|172.16.5.75:2480|2.19GB/64.00GB (3.43%) | |ograph01-5-71(*)|ONLINE|web=NOT_AVAILABLE (MASTER)|31 |09:51:19 |172.16.5.71:2424|172.16.5.71:2480|1.29GB/64.00GB (2.01%) | |ograph04-5-74 |ONLINE|web=ONLINE (REPLICA) |8 |09:52:49 |172.16.5.74:2424|172.16.5.74:2480|896.32MB/64.00GB (1.37%)| |ograph03-5-73 |ONLINE|web=NOT_AVAILABLE (MASTER)|8 |09:52:09 |172.16.5.73:2424|172.16.5.73:2480|1.03GB/64.00GB (1.60%) | |ograph06-5-76 |ONLINE|web=ONLINE (REPLICA) |12 |09:54:10 |172.16.5.76:2424|172.16.5.76:2480|1.84GB/64.00GB (2.88%) | +----------------+------+--------------------------+-----+---------+----------------+----------------+------------------------+ [OHazelcastPlugin] 2020-06-17 09:58:03:678 INFO [ograph01-5-71]<-[ograph03-5-73] Received updated status ograph03-5-73.OSystem=NOT_AVAILABLE [OHazelcastPlugin] 2020-06-17 09:58:07:980 INFO [ograph01-5-71]<-[ograph04-5-74] Received updated status ograph04-5-74.OSystem=NOT_AVAILABLE [OHazelcastPlugin] 2020-06-17 09:58:07:982 INFO [ograph01-5-71]<-[ograph04-5-74] Received updated status ograph04-5-74.web=NOT_AVAILABLE [OHazelcastPlugin] 2020-06-17 09:58:07:983 INFO [ograph01-5-71] Distributed servers status (*=current @=lockmgr[ograph02-5-72]): +----------------+------+---------------------------+-----+---------+----------------+----------------+------------------------+ |Name |Status|Databases |Conns|StartedOn|Binary |HTTP |UsedMemory | +----------------+------+---------------------------+-----+---------+----------------+----------------+------------------------+ |ograph02-5-72(@)|ONLINE|web=NOT_AVAILABLE (MASTER) |10 |09:51:36 |172.16.5.72:2424|172.16.5.72:2480|1.91GB/64.00GB (2.98%) | |ograph05-5-75 |ONLINE|web=ONLINE (REPLICA) |6 |09:53:25 |172.16.5.75:2424|172.16.5.75:2480|2.46GB/64.00GB (3.85%) | |ograph01-5-71(*)|ONLINE|web=NOT_AVAILABLE (MASTER) |31 |09:51:19 |172.16.5.71:2424|172.16.5.71:2480|1.29GB/64.00GB (2.01%) | |ograph04-5-74 |ONLINE|web=NOT_AVAILABLE (REPLICA)|8 |09:52:49 |172.16.5.74:2424|172.16.5.74:2480|896.32MB/64.00GB (1.37%)| |ograph03-5-73 |ONLINE|web=NOT_AVAILABLE (MASTER) |8 |09:52:09 |172.16.5.73:2424|172.16.5.73:2480|1.03GB/64.00GB (1.60%) | |ograph06-5-76 |ONLINE|web=ONLINE (REPLICA) |8 |09:54:10 |172.16.5.76:2424|172.16.5.76:2480|2.10GB/64.00GB (3.28%) | +----------------+------+---------------------------+-----+---------+----------------+----------------+------------------------+ [OHazelcastPlugin] 2020-06-17 09:58:08:800 WARNI [ograph01-5-71] Timeout (165009ms) on waiting for synchronous responses from nodes=[ograph02-5-72, ograph01-5-71, ograph03-5-73] responsesSoFar=[ograph01-5-71, ograph04-5-74, ograph03-5-73, ograph05-5-75, ograph06-5-76] request=(id=0.766 task=TxPhase1 user=#69:0) [ODistributedDatabaseImpl] 2020-06-17 09:58:08:841 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Shutdown request of [172.16.5.74]:2434 is handled [MigrationManager] 2020-06-17 09:58:08:849 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Re-partitioning cluster data... Migration queue size: 307 [MigrationManager] 2020-06-17 09:58:09:349 WARNI [ograph01-5-71] Timeout (84003ms) on waiting for synchronous responses from nodes=[ograph02-5-72, ograph01-5-71, ograph03-5-73] responsesSoFar=[ograph01-5-71, ograph03-5-73, ograph04-5-74, ograph05-5-75, ograph06-5-76] request=(id=0.3606 task=TxPhase1 user=#69:0) [ODistributedDatabaseImpl] 2020-06-17 09:58:09:901 WARNI [ograph01-5-71] Timeout (81004ms) on waiting for synchronous responses from nodes=[ograph02-5-72, ograph01-5-71, ograph03-5-73] responsesSoFar=[ograph01-5-71, ograph03-5-73] request=(id=0.1149 task=TxPhase1 user=#69:0) [ODistributedDatabaseImpl] 2020-06-17 09:58:10:430 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Connection[id=3, /172.16.5.71:2434->/172.16.5.74:43153, endpoint=[172.16.5.74]:2434, alive=false, type=MEMBER] closed. Reason: Connection closed by the other side [TcpIpConnection] 2020-06-17 09:58:10:432 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Connecting to /172.16.5.74:2434, timeout: 0, bind-any: true [TcpIpConnector] 2020-06-17 09:58:10:435 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Could not connect to: /172.16.5.74:2434. Reason: SocketException[Connection refused to address /172.16.5.74:2434] [TcpIpConnector] 2020-06-17 09:58:10:535 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Connecting to /172.16.5.74:2434, timeout: 0, bind-any: true [TcpIpConnector] 2020-06-17 09:58:10:536 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Could not connect to: /172.16.5.74:2434. Reason: SocketException[Connection refused to address /172.16.5.74:2434] [TcpIpConnector] 2020-06-17 09:58:10:636 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Connecting to /172.16.5.74:2434, timeout: 0, bind-any: true [TcpIpConnector] 2020-06-17 09:58:10:636 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Could not connect to: /172.16.5.74:2434. Reason: SocketException[Connection refused to address /172.16.5.74:2434] [TcpIpConnector] 2020-06-17 09:58:10:695 WARNI [ograph01-5-71] Timeout (96004ms) on waiting for synchronous responses from nodes=[ograph02-5-72, ograph01-5-71, ograph03-5-73] responsesSoFar=[ograph01-5-71, ograph03-5-73, ograph05-5-75, ograph04-5-74, ograph06-5-76] request=(id=0.1060 task=TxPhase1 user=#69:0) [ODistributedDatabaseImpl] 2020-06-17 09:58:10:737 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Connecting to /172.16.5.74:2434, timeout: 0, bind-any: true [TcpIpConnector] 2020-06-17 09:58:10:737 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Could not connect to: /172.16.5.74:2434. Reason: SocketException[Connection refused to address /172.16.5.74:2434] [TcpIpConnector] 2020-06-17 09:58:10:750 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Connecting to /172.16.5.74:2434, timeout: 0, bind-any: true [TcpIpConnector] 2020-06-17 09:58:10:751 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Could not connect to: /172.16.5.74:2434. Reason: SocketException[Connection refused to address /172.16.5.74:2434] [TcpIpConnector] 2020-06-17 09:58:10:838 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Connecting to /172.16.5.74:2434, timeout: 0, bind-any: true [TcpIpConnector] 2020-06-17 09:58:10:838 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Could not connect to: /172.16.5.74:2434. Reason: SocketException[Connection refused to address /172.16.5.74:2434] [TcpIpConnector] 2020-06-17 09:58:10:839 WARNI [172.16.5.71]:2434 [orientdb] [3.10.6] Removing connection to endpoint [172.16.5.74]:2434 Cause => java.net.SocketException {Connection refused to address /172.16.5.74:2434}, Error-Count: 5 [TcpIpConnectionErrorHandler] 2020-06-17 09:58:10:839 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Removing Member [172.16.5.74]:2434 - 69f3bf59-4710-4123-a88c-6bf7ef36349c [MembershipManager] 2020-06-17 09:58:10:840 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Members {size:5, ver:7} [ Member [172.16.5.71]:2434 - 111d390c-f6c1-48c3-890b-a004ae134975 this Member [172.16.5.72]:2434 - 6909ffd6-153d-49ce-9286-8defd6aff559 Member [172.16.5.73]:2434 - 6cbb8848-2d8f-44f9-aad9-556bae5e5c55 Member [172.16.5.75]:2434 - 3ce6c7e3-9978-4357-b73d-0040424af433 Member [172.16.5.76]:2434 - 67f4dfe5-ab69-4b5e-9435-33114322a2be ] [ClusterService] 2020-06-17 09:58:10:842 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Committing/rolling-back live transactions of [172.16.5.74]:2434, UUID: 69f3bf59-4710-4123-a88c-6bf7ef36349c [TransactionManagerService] 2020-06-17 09:58:10:843 INFO [ograph01-5-71] Distributed transaction: rolled back 0 transactions and 0 single locks in database 'web' owned by server 'ograph04-5-74' [ODistributedDatabaseImpl] 2020-06-17 09:58:10:845 WARNI [ograph01-5-71] Node removed id=Member [172.16.5.74]:2434 - 69f3bf59-4710-4123-a88c-6bf7ef36349c name=ograph04-5-74 [OHazelcastPlugin] 2020-06-17 09:58:10:845 WARNI [ograph01-5-71] Node 'ograph01-5-71' is NOT_AVAILABLE, the command 'sql.select from OUser where name = ? limit 1' against database 'web' will be executed only on local server with the possibility to have partial result [ODistributedStorage] 2020-06-17 09:58:10:846 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Partition balance is ok, no need to re-partition cluster data... [MigrationManager] 2020-06-17 09:58:10:847 WARNI [ograph01-5-71]->[ograph02-5-72] Error on sending message to distributed node (java.net.SocketException: Connection reset) retrying (1/3) [ORemoteServerChannel] 2020-06-17 09:58:10:848 WARNI [ograph01-5-71]->[ograph02-5-72] Error on reconnecting to distributed node (java.net.ConnectException: Connection refused (Connection refused)) [ORemoteServerChannel] 2020-06-17 09:58:10:848 WARNI [ograph01-5-71]->[ograph02-5-72] Error on sending message to distributed node (java.net.SocketException: Socket closed) retrying (2/3) [ORemoteServerChannel] 2020-06-17 09:58:11:242 WARNI [ograph01-5-71] Timeout (168008ms) on waiting for synchronous responses from nodes=[ograph02-5-72, ograph01-5-71, ograph03-5-73] responsesSoFar=[ograph01-5-71, ograph03-5-73] request=(id=0.655 task=TxPhase1 user=#69:0) [ODistributedDatabaseImpl] 2020-06-17 09:58:11:244 WARNI [ograph01-5-71]->[ograph06-5-76] Error on sending message to distributed node (java.net.SocketException: Broken pipe (Write failed)) retrying (1/3) [ORemoteServerChannel] 2020-06-17 09:58:11:250 WARNI [ograph01-5-71]->[ograph02-5-72] Error on reconnecting to distributed node (java.net.ConnectException: Connection refused (Connection refused)) [ORemoteServerChannel] 2020-06-17 09:58:11:250 WARNI [ograph01-5-71]->[ograph02-5-72] Error on sending message to distributed node (java.net.SocketException: Socket closed) retrying (3/3) [ORemoteServerChannel] 2020-06-17 09:58:11:851 WARNI [ograph01-5-71]->[ograph02-5-72] Error on reconnecting to distributed node (java.net.ConnectException: Connection refused (Connection refused)) [ORemoteServerChannel] 2020-06-17 09:58:11:852 SEVER [ograph01-5-71]->[ograph02-5-72] Error on sending distributed request id=0.3682 task=exc_lock acquire resource=web (err=Connection refused (Connection refused)). Active nodes: [ograph02-5-72, ograph05-5-75, ograph01-5-71, ograph03-5-73, ograph06-5-76] [ODistributedDatabaseImpl] 2020-06-17 09:58:11:860 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] All migration tasks have been completed, queues are empty. [MigrationThread] 2020-06-17 09:58:12:852 SEVER [ograph01-5-71] Cannot re-balance the cluster for database 'web' because the Lock Manager is not available (err=No server active for distributed request (id=0.3682 task=exc_lock acquire resource=web) against database 'OSystem' to nodes [ograph02-5-72]) [OHazelcastPlugin] 2020-06-17 09:58:12:854 WARNI [ograph01-5-71] Timeout (170683ms) on waiting for synchronous responses from nodes=[ograph02-5-72, ograph01-5-71, ograph03-5-73] responsesSoFar=[ograph01-5-71, ograph04-5-74, ograph03-5-73, ograph05-5-75, ograph06-5-76] request=(id=0.468 task=TxPhase1 user=#69:0) [ODistributedDatabaseImpl] 2020-06-17 09:58:12:855 WARNI [ograph01-5-71]->[ograph05-5-75] Error on sending message to distributed node (java.net.SocketException: Connection reset) retrying (1/3) [ORemoteServerChannel] 2020-06-17 09:58:12:866 WARNI [ograph01-5-71]->[ograph06-5-76] Error on sending message to distributed node (java.net.SocketException: Connection reset) retrying (1/3) [ORemoteServerChannel] 2020-06-17 09:58:12:900 WARNI [ograph01-5-71] Timeout (99004ms) on waiting for synchronous responses from nodes=[ograph02-5-72, ograph01-5-71, ograph03-5-73] responsesSoFar=[ograph01-5-71, ograph04-5-74, ograph05-5-75, ograph06-5-76] request=(id=0.840 task=TxPhase1 user=#69:0) [ODistributedDatabaseImpl] 2020-06-17 09:58:12:901 WARNI [ograph01-5-71] Timeout (99003ms) on waiting for synchronous responses from nodes=[ograph02-5-72, ograph01-5-71, ograph03-5-73] responsesSoFar=[ograph01-5-71, ograph03-5-73] request=(id=0.841 task=TxPhase1 user=#69:0) [ODistributedDatabaseImpl] 2020-06-17 09:58:12:902 WARNI [ograph01-5-71] Timeout (99004ms) on waiting for synchronous responses from nodes=[ograph02-5-72, ograph01-5-71, ograph03-5-73] responsesSoFar=[ograph01-5-71, ograph03-5-73] request=(id=0.839 task=TxPhase1 user=#69:0) [ODistributedDatabaseImpl] 2020-06-17 09:58:12:904 WARNI [ograph01-5-71]->[ograph05-5-75] Error on sending message to distributed node (java.net.SocketException: Broken pipe (Write failed)) retrying (1/3) [ORemoteServerChannel] 2020-06-17 09:58:12:904 WARNI [ograph01-5-71]->[ograph06-5-76] Error on sending message to distributed node (java.net.SocketException: Broken pipe (Write failed)) retrying (1/3) [ORemoteServerChannel]Cannot determine protocol version for server ograph02-5-72 error: Connection refused (Connection refused) java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at com.orientechnologies.orient.client.binary.OChannelBinaryClientAbstract.(OChannelBinaryClientAbstract.java:70) at com.orientechnologies.orient.client.binary.OChannelBinarySynchClient.(OChannelBinarySynchClient.java:32) at com.orientechnologies.orient.server.distributed.ORemoteServerChannel.connect(ORemoteServerChannel.java:122) at com.orientechnologies.orient.server.distributed.ORemoteServerChannel.(ORemoteServerChannel.java:90) at com.orientechnologies.orient.server.distributed.ORemoteServerController.(ORemoteServerController.java:53) at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.getRemoteServer(OHazelcastPlugin.java:683) at com.orientechnologies.orient.server.distributed.impl.task.ORemoteTaskFactoryManagerImpl.getFactoryByServerName(ORemoteTaskFactoryManagerImpl.java:74) at com.orientechnologies.orient.server.distributed.impl.task.ORemoteTaskFactoryManagerImpl.getFactoryByServerNames(ORemoteTaskFactoryManagerImpl.java:61) at com.orientechnologies.orient.server.distributed.impl.ONewDistributedTransactionManager.createTxTask(ONewDistributedTransactionManager.java:383) at com.orientechnologies.orient.server.distributed.impl.ONewDistributedTransactionManager.retriedCommit(ONewDistributedTransactionManager.java:139) at com.orientechnologies.orient.server.distributed.impl.ONewDistributedTransactionManager.commit(ONewDistributedTransactionManager.java:84) at com.orientechnologies.orient.server.distributed.impl.ODatabaseDocumentDistributed.internalCommit(ODatabaseDocumentDistributed.java:346) at com.orientechnologies.orient.core.tx.OTransactionOptimistic.doCommit(OTransactionOptimistic.java:533) at com.orientechnologies.orient.core.tx.OTransactionOptimistic.commit(OTransactionOptimistic.java:103) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentAbstract.commit(ODatabaseDocumentAbstract.java:2183) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentAbstract.commit(ODatabaseDocumentAbstract.java:2153) at com.orientechnologies.orient.server.OConnectionBinaryExecutor.executeCommit37(OConnectionBinaryExecutor.java:1282) at com.orientechnologies.orient.client.remote.message.OCommit37Request.execute(OCommit37Request.java:124) at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.sessionRequest(ONetworkProtocolBinary.java:317) at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.execute(ONetworkProtocolBinary.java:212) at com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:69) Cannot determine protocol version for server ograph02-5-72 error: Connection refused (Connection refused) java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at com.orientechnologies.orient.client.binary.OChannelBinaryClientAbstract.(OChannelBinaryClientAbstract.java:70) at com.orientechnologies.orient.client.binary.OChannelBinarySynchClient.(OChannelBinarySynchClient.java:32) at com.orientechnologies.orient.server.distributed.ORemoteServerChannel.connect(ORemoteServerChannel.java:122) at com.orientechnologies.orient.server.distributed.ORemoteServerChannel.(ORemoteServerChannel.java:90) at com.orientechnologies.orient.server.distributed.ORemoteServerController.(ORemoteServerController.java:53) at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.getRemoteServer(OHazelcastPlugin.java:683) at com.orientechnologies.orient.server.distributed.impl.task.ORemoteTaskFactoryManagerImpl.getFactoryByServerName(ORemoteTaskFactoryManagerImpl.java:74) at com.orientechnologies.orient.server.distributed.impl.task.ORemoteTaskFactoryManagerImpl.getFactoryByServerNames(ORemoteTaskFactoryManagerImpl.java:61) at com.orientechnologies.orient.server.distributed.impl.ONewDistributedTransactionManager.createTxTask(ONewDistributedTransactionManager.java:383) at com.orientechnologies.orient.server.distributed.impl.ONewDistributedTransactionManager.retriedCommit(ONewDistributedTransactionManager.java:139) at com.orientechnologies.orient.server.distributed.impl.ONewDistributedTransactionManager.commit(ONewDistributedTransactionManager.java:84) at com.orientechnologies.orient.server.distributed.impl.ODatabaseDocumentDistributed.internalCommit(ODatabaseDocumentDistributed.java:346) at com.orientechnologies.orient.core.tx.OTransactionOptimistic.doCommit(OTransactionOptimistic.java:533) at com.orientechnologies.orient.core.tx.OTransactionOptimistic.commit(OTransactionOptimistic.java:103) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentAbstract.commit(ODatabaseDocumentAbstract.java:2183) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentAbstract.commit(ODatabaseDocumentAbstract.java:2153) at com.orientechnologies.orient.server.OConnectionBinaryExecutor.executeCommit37(OConnectionBinaryExecutor.java:1282) at com.orientechnologies.orient.client.remote.message.OCommit37Request.execute(OCommit37Request.java:124) at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.sessionRequest(ONetworkProtocolBinary.java:317) at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.execute(ONetworkProtocolBinary.java:212) at com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:69) 2020-06-17 09:58:12:916 WARNI [ograph01-5-71]->[ograph06-5-76] Error on sending message to distributed node (java.net.SocketException: Connection reset) retrying (1/3) [ORemoteServerChannel] 2020-06-17 09:58:12:918 INFO [ograph01-5-71]->[ograph03-5-73] Elected server 'ograph03-5-73' as new lockManager (old=ograph02-5-72) [OHazelcastPlugin] 2020-06-17 09:58:12:922 WARNI [ograph01-5-71] Node removed id=Member [172.16.5.72]:2434 - 6909ffd6-153d-49ce-9286-8defd6aff559 name=ograph02-5-72 [OHazelcastPlugin] 2020-06-17 09:58:12:923 INFO [ograph01-5-71] Distributed transaction: rolled back 0 transactions and 0 single locks in database 'web' owned by server 'ograph02-5-72' [ODistributedDatabaseImpl] 2020-06-17 09:58:12:923 WARNI [ograph01-5-71] Node 'ograph01-5-71' is NOT_AVAILABLE, the command 'sql.select from OUser where name = ? limit 1' against database 'web' will be executed only on local server with the possibility to have partial result [ODistributedStorage] 2020-06-17 09:58:13:070 WARNI [ograph01-5-71] Timeout (171008ms) on waiting for synchronous responses from nodes=[ograph02-5-72, ograph01-5-71, ograph03-5-73] responsesSoFar=[ograph01-5-71, ograph05-5-75, ograph04-5-74, ograph06-5-76] request=(id=0.446 task=TxPhase1 user=#69:0) [ODistributedDatabaseImpl] 2020-06-17 09:58:13:071 WARNI [ograph01-5-71]->[ograph05-5-75] Error on sending message to distributed node (java.net.SocketException: Connection reset) retrying (1/3) [ORemoteServerChannel] 2020-06-17 09:58:13:075 WARNI [ograph01-5-71]->[ograph06-5-76] Error on sending message to distributed node (java.net.SocketException: Connection reset) retrying (1/3) [ORemoteServerChannel] 2020-06-17 09:58:13:080 SEVER [ograph01-5-71]->[ograph02-5-72] Error on connecting to node ograph02-5-72 [OHazelcastPlugin] 2020-06-17 09:58:13:081 INFO [ograph01-5-71]<-[ograph02-5-72] Added node configuration id=Member [172.16.5.72]:2434 - 6909ffd6-153d-49ce-9286-8defd6aff559 name=ograph02-5-72, now 5 nodes are configured [OHazelcastPlugin] 2020-06-17 09:58:13:081 INFO [ograph01-5-71] Distributed servers status (*=current @=lockmgr[ograph03-5-73]): +----------------+------+--------------------------+-----+---------+----------------+----------------+----------------------+ |Name |Status|Databases |Conns|StartedOn|Binary |HTTP |UsedMemory | +----------------+------+--------------------------+-----+---------+----------------+----------------+----------------------+ |ograph02-5-72 |ONLINE|web=NOT_AVAILABLE (MASTER)|10 |09:51:36 |172.16.5.72:2424|172.16.5.72:2480|1.91GB/64.00GB (2.98%)| |ograph05-5-75 |ONLINE|web=ONLINE (REPLICA) |6 |09:53:25 |172.16.5.75:2424|172.16.5.75:2480|2.46GB/64.00GB (3.85%)| |ograph01-5-71(*)|ONLINE|web=NOT_AVAILABLE (MASTER)|31 |09:51:19 |172.16.5.71:2424|172.16.5.71:2480|1.30GB/64.00GB (2.03%)| |ograph03-5-73(@)|ONLINE|web=NOT_AVAILABLE (MASTER)|8 |09:52:09 |172.16.5.73:2424|172.16.5.73:2480|1.03GB/64.00GB (1.60%)| |ograph06-5-76 |ONLINE|web=ONLINE (REPLICA) |8 |09:54:10 |172.16.5.76:2424|172.16.5.76:2480|2.10GB/64.00GB (3.28%)| +----------------+------+--------------------------+-----+---------+----------------+----------------+----------------------+ [OHazelcastPlugin]Cannot determine protocol version for server ograph02-5-72 error: Connection refused (Connection refused) java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at com.orientechnologies.orient.client.binary.OChannelBinaryClientAbstract.(OChannelBinaryClientAbstract.java:70) at com.orientechnologies.orient.client.binary.OChannelBinarySynchClient.(OChannelBinarySynchClient.java:32) at com.orientechnologies.orient.server.distributed.ORemoteServerChannel.connect(ORemoteServerChannel.java:122) at com.orientechnologies.orient.server.distributed.ORemoteServerChannel.(ORemoteServerChannel.java:90) at com.orientechnologies.orient.server.distributed.ORemoteServerController.(ORemoteServerController.java:53) at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.getRemoteServer(OHazelcastPlugin.java:683) at com.orientechnologies.orient.server.distributed.impl.task.ORemoteTaskFactoryManagerImpl.getFactoryByServerName(ORemoteTaskFactoryManagerImpl.java:74) at com.orientechnologies.orient.server.distributed.impl.task.ORemoteTaskFactoryManagerImpl.getFactoryByServerNames(ORemoteTaskFactoryManagerImpl.java:61) at com.orientechnologies.orient.server.distributed.impl.ONewDistributedTransactionManager.createTxTask(ONewDistributedTransactionManager.java:383) at com.orientechnologies.orient.server.distributed.impl.ONewDistributedTransactionManager.retriedCommit(ONewDistributedTransactionManager.java:139) at com.orientechnologies.orient.server.distributed.impl.ONewDistributedTransactionManager.commit(ONewDistributedTransactionManager.java:84) at com.orientechnologies.orient.server.distributed.impl.ODatabaseDocumentDistributed.internalCommit(ODatabaseDocumentDistributed.java:346) at com.orientechnologies.orient.core.tx.OTransactionOptimistic.doCommit(OTransactionOptimistic.java:533) at com.orientechnologies.orient.core.tx.OTransactionOptimistic.commit(OTransactionOptimistic.java:103) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentAbstract.commit(ODatabaseDocumentAbstract.java:2183) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentAbstract.commit(ODatabaseDocumentAbstract.java:2153) at com.orientechnologies.orient.server.OConnectionBinaryExecutor.executeCommit37(OConnectionBinaryExecutor.java:1282) at com.orientechnologies.orient.client.remote.message.OCommit37Request.execute(OCommit37Request.java:124) at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.sessionRequest(ONetworkProtocolBinary.java:317) at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.execute(ONetworkProtocolBinary.java:212) at com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:69) 2020-06-17 09:58:13:083 WARNI [ograph01-5-71] Node removed id=Member [172.16.5.72]:2434 - 6909ffd6-153d-49ce-9286-8defd6aff559 name=ograph02-5-72 [OHazelcastPlugin] 2020-06-17 09:58:13:083 INFO [ograph01-5-71] Distributed transaction: rolled back 0 transactions and 0 single locks in database 'web' owned by server 'ograph02-5-72' [ODistributedDatabaseImpl] 2020-06-17 09:58:13:083 WARNI [ograph01-5-71] Node 'ograph01-5-71' is NOT_AVAILABLE, the command 'sql.select from OUser where name = ? limit 1' against database 'web' will be executed only on local server with the possibility to have partial result [ODistributedStorage] 2020-06-17 09:58:13:084 WARNI [ograph01-5-71]->[ograph03-5-73] Error on sending message to distributed node (java.net.SocketException: Broken pipe (Write failed)) retrying (1/3) [ORemoteServerChannel] 2020-06-17 09:58:13:085 WARNI [ograph01-5-71]->[ograph03-5-73] Error on reconnecting to distributed node (java.net.ConnectException: Connection refused (Connection refused)) [ORemoteServerChannel] 2020-06-17 09:58:13:085 WARNI [ograph01-5-71]->[ograph03-5-73] Error on sending message to distributed node (java.net.SocketException: Socket closed) retrying (2/3) [ORemoteServerChannel] 2020-06-17 09:58:13:139 WARNI [ograph01-5-71] Timeout (171006ms) on waiting for synchronous responses from nodes=[ograph02-5-72, ograph01-5-71, ograph03-5-73] responsesSoFar=[ograph01-5-71, ograph03-5-73] request=(id=0.459 task=TxPhase1 user=#69:0) [ODistributedDatabaseImpl] 2020-06-17 09:58:13:473 WARNI [ograph01-5-71] Timeout (99004ms) on waiting for synchronous responses from nodes=[ograph02-5-72, ograph01-5-71, ograph03-5-73] responsesSoFar=[ograph01-5-71] request=(id=0.990 task=TxPhase1 user=#69:0) [ODistributedDatabaseImpl] 2020-06-17 09:58:13:473 WARNI [ograph01-5-71]->[ograph05-5-75] Error on sending message to distributed node (java.net.SocketException: Broken pipe (Write failed)) retrying (1/3) [ORemoteServerChannel] 2020-06-17 09:58:13:477 WARNI [ograph01-5-71]->[ograph06-5-76] Error on sending message to distributed node (java.net.SocketException: Broken pipe (Write failed)) retrying (1/3) [ORemoteServerChannel] 2020-06-17 09:58:13:486 WARNI [ograph01-5-71]->[ograph03-5-73] Error on reconnecting to distributed node (java.net.ConnectException: Connection refused (Connection refused)) [ORemoteServerChannel] 2020-06-17 09:58:13:486 WARNI [ograph01-5-71]->[ograph03-5-73] Error on sending message to distributed node (java.net.SocketException: Socket closed) retrying (3/3) [ORemoteServerChannel] 2020-06-17 09:58:14:035 INFO [ograph01-5-71]<-[ograph05-5-75] Received updated status ograph05-5-75.web=NOT_AVAILABLE [OHazelcastPlugin] 2020-06-17 09:58:14:036 INFO [ograph01-5-71] Distributed servers status (*=current @=lockmgr[ograph03-5-73]): +----------------+------+---------------------------+-----+---------+----------------+----------------+----------------------+ |Name |Status|Databases |Conns|StartedOn|Binary |HTTP |UsedMemory | +----------------+------+---------------------------+-----+---------+----------------+----------------+----------------------+ |ograph05-5-75 |ONLINE|web=NOT_AVAILABLE (REPLICA)|6 |09:53:25 |172.16.5.75:2424|172.16.5.75:2480|2.46GB/64.00GB (3.85%)| |ograph01-5-71(*)|ONLINE|web=NOT_AVAILABLE (MASTER) |31 |09:51:19 |172.16.5.71:2424|172.16.5.71:2480|1.30GB/64.00GB (2.03%)| |ograph03-5-73(@)|ONLINE|web=NOT_AVAILABLE (MASTER) |8 |09:52:09 |172.16.5.73:2424|172.16.5.73:2480|1.03GB/64.00GB (1.60%)| |ograph06-5-76 |ONLINE|web=ONLINE (REPLICA) |8 |09:54:10 |172.16.5.76:2424|172.16.5.76:2480|2.10GB/64.00GB (3.28%)| +----------------+------+---------------------------+-----+---------+----------------+----------------+----------------------+ [OHazelcastPlugin] 2020-06-17 09:58:14:037 INFO [ograph01-5-71]<-[ograph05-5-75] Received updated status ograph05-5-75.OSystem=NOT_AVAILABLE [OHazelcastPlugin] 2020-06-17 09:58:14:087 WARNI [ograph01-5-71]->[ograph03-5-73] Error on reconnecting to distributed node (java.net.ConnectException: Connection refused (Connection refused)) [ORemoteServerChannel] 2020-06-17 09:58:14:087 SEVER [ograph01-5-71]->[ograph03-5-73] Error on sending distributed request id=0.3694 task=exc_lock acquire resource=web (err=Connection refused (Connection refused)). Active nodes: [ograph05-5-75, ograph01-5-71, ograph03-5-73, ograph06-5-76] [ODistributedDatabaseImpl] 2020-06-17 09:58:14:966 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Shutdown request of [172.16.5.75]:2434 is handled [MigrationManager] 2020-06-17 09:58:14:971 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Re-partitioning cluster data... Migration queue size: 236 [MigrationManager] 2020-06-17 09:58:15:088 SEVER [ograph01-5-71] Cannot re-balance the cluster for database 'web' because the Lock Manager is not available (err=No server active for distributed request (id=0.3694 task=exc_lock acquire resource=web) against database 'OSystem' to nodes [ograph03-5-73]) [OHazelcastPlugin]Cannot determine protocol version for server ograph03-5-73 error: Connection refused (Connection refused) java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at com.orientechnologies.orient.client.binary.OChannelBinaryClientAbstract.(OChannelBinaryClientAbstract.java:70) at com.orientechnologies.orient.client.binary.OChannelBinarySynchClient.(OChannelBinarySynchClient.java:32) at com.orientechnologies.orient.server.distributed.ORemoteServerChannel.connect(ORemoteServerChannel.java:122) at com.orientechnologies.orient.server.distributed.ORemoteServerChannel.(ORemoteServerChannel.java:90) at com.orientechnologies.orient.server.distributed.ORemoteServerController.(ORemoteServerController.java:53) at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.getRemoteServer(OHazelcastPlugin.java:683) at com.orientechnologies.orient.server.distributed.impl.task.ORemoteTaskFactoryManagerImpl.getFactoryByServerName(ORemoteTaskFactoryManagerImpl.java:74) at com.orientechnologies.orient.server.distributed.impl.task.ORemoteTaskFactoryManagerImpl.getFactoryByServerNames(ORemoteTaskFactoryManagerImpl.java:61) at com.orientechnologies.orient.server.distributed.impl.ONewDistributedTransactionManager.createTxTask(ONewDistributedTransactionManager.java:383) at com.orientechnologies.orient.server.distributed.impl.ONewDistributedTransactionManager.retriedCommit(ONewDistributedTransactionManager.java:139) at com.orientechnologies.orient.server.distributed.impl.ONewDistributedTransactionManager.commit(ONewDistributedTransactionManager.java:84) at com.orientechnologies.orient.server.distributed.impl.ODatabaseDocumentDistributed.internalCommit(ODatabaseDocumentDistributed.java:346) at com.orientechnologies.orient.core.tx.OTransactionOptimistic.doCommit(OTransactionOptimistic.java:533) at com.orientechnologies.orient.core.tx.OTransactionOptimistic.commit(OTransactionOptimistic.java:103) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentAbstract.commit(ODatabaseDocumentAbstract.java:2183) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentAbstract.commit(ODatabaseDocumentAbstract.java:2153) at com.orientechnologies.orient.server.OConnectionBinaryExecutor.executeCommit37(OConnectionBinaryExecutor.java:1282) at com.orientechnologies.orient.client.remote.message.OCommit37Request.execute(OCommit37Request.java:124) at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.sessionRequest(ONetworkProtocolBinary.java:317) at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.execute(ONetworkProtocolBinary.java:212) at com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:69) 2020-06-17 09:58:15:098 INFO [ograph01-5-71]->[ograph05-5-75] Elected server 'ograph05-5-75' as new lockManager (old=ograph03-5-73) [OHazelcastPlugin] 2020-06-17 09:58:15:099 WARNI [ograph01-5-71] Node removed id=Member [172.16.5.73]:2434 - 6cbb8848-2d8f-44f9-aad9-556bae5e5c55 name=ograph03-5-73 [OHazelcastPlugin] 2020-06-17 09:58:15:099 INFO [ograph01-5-71] Distributed transaction: rolled back 0 transactions and 0 single locks in database 'web' owned by server 'ograph03-5-73' [ODistributedDatabaseImpl] 2020-06-17 09:58:15:100 WARNI [ograph01-5-71] Node 'ograph01-5-71' is NOT_AVAILABLE, the command 'sql.select from OUser where name = ? limit 1' against database 'web' will be executed only on local server with the possibility to have partial result [ODistributedStorage] 2020-06-17 09:58:16:005 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Connection[id=4, /172.16.5.71:2434->/172.16.5.75:55483, endpoint=[172.16.5.75]:2434, alive=false, type=MEMBER] closed. Reason: Connection closed by the other side [TcpIpConnection] 2020-06-17 09:58:16:006 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Connecting to /172.16.5.75:2434, timeout: 0, bind-any: true [TcpIpConnector] 2020-06-17 09:58:16:006 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Could not connect to: /172.16.5.75:2434. Reason: SocketException[Connection refused to address /172.16.5.75:2434] [TcpIpConnector] 2020-06-17 09:58:16:107 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Connecting to /172.16.5.75:2434, timeout: 0, bind-any: true [TcpIpConnector] 2020-06-17 09:58:16:108 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Could not connect to: /172.16.5.75:2434. Reason: SocketException[Connection refused to address /172.16.5.75:2434] [TcpIpConnector] 2020-06-17 09:58:16:208 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Connecting to /172.16.5.75:2434, timeout: 0, bind-any: true [TcpIpConnector] 2020-06-17 09:58:16:210 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Could not connect to: /172.16.5.75:2434. Reason: SocketException[Connection refused to address /172.16.5.75:2434] [TcpIpConnector] 2020-06-17 09:58:16:310 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Connecting to /172.16.5.75:2434, timeout: 0, bind-any: true [TcpIpConnector] 2020-06-17 09:58:16:311 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Could not connect to: /172.16.5.75:2434. Reason: SocketException[Connection refused to address /172.16.5.75:2434] [TcpIpConnector] 2020-06-17 09:58:16:311 WARNI [172.16.5.71]:2434 [orientdb] [3.10.6] Removing connection to endpoint [172.16.5.75]:2434 Cause => java.net.SocketException {Connection refused to address /172.16.5.75:2434}, Error-Count: 5 [TcpIpConnectionErrorHandler] 2020-06-17 09:58:16:312 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Removing Member [172.16.5.75]:2434 - 3ce6c7e3-9978-4357-b73d-0040424af433 [MembershipManager] 2020-06-17 09:58:16:313 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Members {size:4, ver:8} [ Member [172.16.5.71]:2434 - 111d390c-f6c1-48c3-890b-a004ae134975 this Member [172.16.5.72]:2434 - 6909ffd6-153d-49ce-9286-8defd6aff559 Member [172.16.5.73]:2434 - 6cbb8848-2d8f-44f9-aad9-556bae5e5c55 Member [172.16.5.76]:2434 - 67f4dfe5-ab69-4b5e-9435-33114322a2be ] [ClusterService] 2020-06-17 09:58:16:314 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Committing/rolling-back live transactions of [172.16.5.75]:2434, UUID: 3ce6c7e3-9978-4357-b73d-0040424af433 [TransactionManagerService] 2020-06-17 09:58:16:317 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Partition balance is ok, no need to re-partition cluster data... [MigrationManager] 2020-06-17 09:58:16:329 WARNI [ograph01-5-71] Node removed id=Member [172.16.5.75]:2434 - 3ce6c7e3-9978-4357-b73d-0040424af433 name=ograph05-5-75 [OHazelcastPlugin] 2020-06-17 09:58:16:329 INFO [ograph01-5-71] Distributed transaction: rolled back 0 transactions and 0 single locks in database 'web' owned by server 'ograph05-5-75' [ODistributedDatabaseImpl] 2020-06-17 09:58:16:330 WARNI [ograph01-5-71] Node 'ograph01-5-71' is NOT_AVAILABLE, the command 'sql.select from OUser where name = ? limit 1' against database 'web' will be executed only on local server with the possibility to have partial result [ODistributedStorage] 2020-06-17 09:58:16:331 WARNI [ograph01-5-71]->[ograph06-5-76] Error on sending message to distributed node (java.net.SocketException: Broken pipe (Write failed)) retrying (1/3) [ORemoteServerChannel] 2020-06-17 09:58:17:324 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] All migration tasks have been completed, queues are empty. [MigrationThread] 2020-06-17 09:58:17:536 INFO [ograph01-5-71] Distributed servers status (*=current @=lockmgr[ograph06-5-76]): +----------------+------+--------------------------+-----+---------+----------------+----------------+----------------------+ |Name |Status|Databases |Conns|StartedOn|Binary |HTTP |UsedMemory | +----------------+------+--------------------------+-----+---------+----------------+----------------+----------------------+ |ograph01-5-71(*)|ONLINE|web=NOT_AVAILABLE (MASTER)|31 |09:51:19 |172.16.5.71:2424|172.16.5.71:2480|1.30GB/64.00GB (2.03%)| |ograph06-5-76(@)|ONLINE|web=ONLINE (REPLICA) |3 |09:54:10 |172.16.5.76:2424|172.16.5.76:2480|2.27GB/64.00GB (3.55%)| +----------------+------+--------------------------+-----+---------+----------------+----------------+----------------------+ [OHazelcastPlugin] 2020-06-17 09:58:18:699 INFO [ograph01-5-71]<-[ograph06-5-76] Received updated status ograph06-5-76.web=NOT_AVAILABLE [OHazelcastPlugin] 2020-06-17 09:58:18:700 INFO [ograph01-5-71] Distributed servers status (*=current @=lockmgr[ograph06-5-76]): +----------------+------+---------------------------+-----+---------+----------------+----------------+----------------------+ |Name |Status|Databases |Conns|StartedOn|Binary |HTTP |UsedMemory | +----------------+------+---------------------------+-----+---------+----------------+----------------+----------------------+ |ograph01-5-71(*)|ONLINE|web=NOT_AVAILABLE (MASTER) |31 |09:51:19 |172.16.5.71:2424|172.16.5.71:2480|1.30GB/64.00GB (2.03%)| |ograph06-5-76(@)|ONLINE|web=NOT_AVAILABLE (REPLICA)|3 |09:54:10 |172.16.5.76:2424|172.16.5.76:2480|2.27GB/64.00GB (3.55%)| +----------------+------+---------------------------+-----+---------+----------------+----------------+----------------------+ [OHazelcastPlugin] 2020-06-17 09:58:18:700 INFO [ograph01-5-71]<-[ograph06-5-76] Received updated status ograph06-5-76.OSystem=NOT_AVAILABLE [OHazelcastPlugin] 2020-06-17 09:58:19:561 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Shutdown request of [172.16.5.76]:2434 is handled [MigrationManager] 2020-06-17 09:58:19:568 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Re-partitioning cluster data... Migration queue size: 195 [MigrationManager] 2020-06-17 09:58:20:285 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Connection[id=5, /172.16.5.71:2434->/172.16.5.76:41063, endpoint=[172.16.5.76]:2434, alive=false, type=MEMBER] closed. Reason: Connection closed by the other side [TcpIpConnection] 2020-06-17 09:58:20:286 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Connecting to /172.16.5.76:2434, timeout: 0, bind-any: true [TcpIpConnector] 2020-06-17 09:58:20:287 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Could not connect to: /172.16.5.76:2434. Reason: SocketException[Connection refused to address /172.16.5.76:2434] [TcpIpConnector] 2020-06-17 09:58:20:387 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Connecting to /172.16.5.76:2434, timeout: 0, bind-any: true [TcpIpConnector] 2020-06-17 09:58:20:388 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Could not connect to: /172.16.5.76:2434. Reason: SocketException[Connection refused to address /172.16.5.76:2434] [TcpIpConnector] 2020-06-17 09:58:20:488 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Connecting to /172.16.5.76:2434, timeout: 0, bind-any: true [TcpIpConnector] 2020-06-17 09:58:20:489 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Could not connect to: /172.16.5.76:2434. Reason: SocketException[Connection refused to address /172.16.5.76:2434] [TcpIpConnector] 2020-06-17 09:58:20:589 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Connecting to /172.16.5.76:2434, timeout: 0, bind-any: true [TcpIpConnector] 2020-06-17 09:58:20:590 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Could not connect to: /172.16.5.76:2434. Reason: SocketException[Connection refused to address /172.16.5.76:2434] [TcpIpConnector] 2020-06-17 09:58:20:590 WARNI [172.16.5.71]:2434 [orientdb] [3.10.6] Removing connection to endpoint [172.16.5.76]:2434 Cause => java.net.SocketException {Connection refused to address /172.16.5.76:2434}, Error-Count: 5 [TcpIpConnectionErrorHandler] 2020-06-17 09:58:20:590 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Removing Member [172.16.5.76]:2434 - 67f4dfe5-ab69-4b5e-9435-33114322a2be [MembershipManager] 2020-06-17 09:58:20:591 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Committing/rolling-back live transactions of [172.16.5.76]:2434, UUID: 67f4dfe5-ab69-4b5e-9435-33114322a2be [TransactionManagerService] 2020-06-17 09:58:20:592 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Members {size:3, ver:9} [ Member [172.16.5.71]:2434 - 111d390c-f6c1-48c3-890b-a004ae134975 this Member [172.16.5.72]:2434 - 6909ffd6-153d-49ce-9286-8defd6aff559 Member [172.16.5.73]:2434 - 6cbb8848-2d8f-44f9-aad9-556bae5e5c55 ] [ClusterService] 2020-06-17 09:58:20:595 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Partition balance is ok, no need to re-partition cluster data... [MigrationManager] 2020-06-17 09:58:20:600 WARNI [ograph01-5-71] Node removed id=Member [172.16.5.76]:2434 - 67f4dfe5-ab69-4b5e-9435-33114322a2be name=ograph06-5-76 [OHazelcastPlugin] 2020-06-17 09:58:20:600 INFO [ograph01-5-71] Distributed transaction: rolled back 0 transactions and 0 single locks in database 'web' owned by server 'ograph06-5-76' [ODistributedDatabaseImpl] 2020-06-17 09:58:20:601 WARNI [ograph01-5-71] Node 'ograph01-5-71' is NOT_AVAILABLE, the command 'sql.select from OUser where name = ? limit 1' against database 'web' will be executed only on local server with the possibility to have partial result [ODistributedStorage] 2020-06-17 09:58:20:607 INFO [ograph01-5-71] Reassigning ownership of clusters for database web... [OHazelcastPlugin] 2020-06-17 09:58:20:607 INFO [ograph01-5-71] Reassignment of clusters for database 'web' completed (classes=167) [OHazelcastPlugin] 2020-06-17 09:58:20:610 WARNI [ograph01-5-71] No undo because the task (exc_lock acquire resource=web) is idempotent [ODistributedResponseManagerImpl] 2020-06-17 09:58:20:610 WARNI [ograph01-5-71] Server 'ograph01-5-71' cannot acquire distributed lock on resource 'web' (timeout=20000)... [ODistributedLockManagerRequester] 2020-06-17 09:58:20:610 SEVER [ograph01-5-71] Cannot re-balance the cluster for database 'web' because the Lock Manager is not available (err=Server 'ograph01-5-71' cannot acquire exclusive lock on resource 'web' (timeout=20000)) [OHazelcastPlugin] 2020-06-17 09:58:20:612 WARNI [ograph01-5-71] No undo because the task (exc_lock acquire resource=web) is idempotent [ODistributedResponseManagerImpl] 2020-06-17 09:58:20:612 WARNI [ograph01-5-71] Server 'ograph01-5-71' cannot acquire distributed lock on resource 'web' (timeout=20000)... [ODistributedLockManagerRequester] 2020-06-17 09:58:20:613 SEVER [ograph01-5-71] Cannot re-balance the cluster for database 'web' because the Lock Manager is not available (err=Server 'ograph01-5-71' cannot acquire exclusive lock on resource 'web' (timeout=20000)) [OHazelcastPlugin] 2020-06-17 09:58:20:615 WARNI [ograph01-5-71] No undo because the task (exc_lock acquire resource=web) is idempotent [ODistributedResponseManagerImpl] 2020-06-17 09:58:20:615 WARNI [ograph01-5-71] Server 'ograph01-5-71' cannot acquire distributed lock on resource 'web' (timeout=20000)... [ODistributedLockManagerRequester] 2020-06-17 09:58:20:615 SEVER [ograph01-5-71] Cannot re-balance the cluster for database 'web' because the Lock Manager is not available (err=Server 'ograph01-5-71' cannot acquire exclusive lock on resource 'web' (timeout=20000)) [OHazelcastPlugin] 2020-06-17 09:58:20:618 SEVER [ograph01-5-71]->[ograph03-5-73] Error on connecting to node ograph03-5-73 [OHazelcastPlugin] 2020-06-17 09:58:20:618 INFO [ograph01-5-71]<-[ograph03-5-73] Added node configuration id=Member [172.16.5.73]:2434 - 6cbb8848-2d8f-44f9-aad9-556bae5e5c55 name=ograph03-5-73, now 2 nodes are configured [OHazelcastPlugin] 2020-06-17 09:58:20:618 INFO [ograph01-5-71] Distributed servers status (*=current @=lockmgr[ograph01-5-71]): +-------------------+------+--------------------------+-----+---------+----------------+----------------+----------------------+ |Name |Status|Databases |Conns|StartedOn|Binary |HTTP |UsedMemory | +-------------------+------+--------------------------+-----+---------+----------------+----------------+----------------------+ |ograph01-5-71(*)(@)|ONLINE|web=NOT_AVAILABLE (MASTER)|31 |09:51:19 |172.16.5.71:2424|172.16.5.71:2480|1.45GB/64.00GB (2.27%)| |ograph03-5-73 |ONLINE|web=NOT_AVAILABLE (MASTER)|8 |09:52:09 |172.16.5.73:2424|172.16.5.73:2480|1.03GB/64.00GB (1.60%)| +-------------------+------+--------------------------+-----+---------+----------------+----------------+----------------------+ [OHazelcastPlugin]Cannot determine protocol version for server ograph03-5-73 error: Connection refused (Connection refused) java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at com.orientechnologies.orient.client.binary.OChannelBinaryClientAbstract.(OChannelBinaryClientAbstract.java:70) at com.orientechnologies.orient.client.binary.OChannelBinarySynchClient.(OChannelBinarySynchClient.java:32) at com.orientechnologies.orient.server.distributed.ORemoteServerChannel.connect(ORemoteServerChannel.java:122) at com.orientechnologies.orient.server.distributed.ORemoteServerChannel.(ORemoteServerChannel.java:90) at com.orientechnologies.orient.server.distributed.ORemoteServerController.(ORemoteServerController.java:53) at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.getRemoteServer(OHazelcastPlugin.java:683) at com.orientechnologies.orient.server.distributed.impl.task.ORemoteTaskFactoryManagerImpl.getFactoryByServerName(ORemoteTaskFactoryManagerImpl.java:74) at com.orientechnologies.orient.server.distributed.impl.task.ORemoteTaskFactoryManagerImpl.getFactoryByServerNames(ORemoteTaskFactoryManagerImpl.java:61) at com.orientechnologies.orient.server.distributed.impl.ONewDistributedTransactionManager.createTxTask(ONewDistributedTransactionManager.java:383) at com.orientechnologies.orient.server.distributed.impl.ONewDistributedTransactionManager.retriedCommit(ONewDistributedTransactionManager.java:139) at com.orientechnologies.orient.server.distributed.impl.ONewDistributedTransactionManager.commit(ONewDistributedTransactionManager.java:84) at com.orientechnologies.orient.server.distributed.impl.ODatabaseDocumentDistributed.internalCommit(ODatabaseDocumentDistributed.java:346) at com.orientechnologies.orient.core.tx.OTransactionOptimistic.doCommit(OTransactionOptimistic.java:533) at com.orientechnologies.orient.core.tx.OTransactionOptimistic.commit(OTransactionOptimistic.java:103) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentAbstract.commit(ODatabaseDocumentAbstract.java:2183) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentAbstract.commit(ODatabaseDocumentAbstract.java:2153) at com.orientechnologies.orient.server.OConnectionBinaryExecutor.executeCommit37(OConnectionBinaryExecutor.java:1282) at com.orientechnologies.orient.client.remote.message.OCommit37Request.execute(OCommit37Request.java:124) at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.sessionRequest(ONetworkProtocolBinary.java:317) at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.execute(ONetworkProtocolBinary.java:212) at com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:69) 2020-06-17 09:58:20:620 WARNI [ograph01-5-71] Node removed id=Member [172.16.5.73]:2434 - 6cbb8848-2d8f-44f9-aad9-556bae5e5c55 name=ograph03-5-73 [OHazelcastPlugin] 2020-06-17 09:58:20:620 INFO [ograph01-5-71] Distributed transaction: rolled back 0 transactions and 0 single locks in database 'web' owned by server 'ograph03-5-73' [ODistributedDatabaseImpl] 2020-06-17 09:58:20:620 WARNI [ograph01-5-71] Node 'ograph01-5-71' is NOT_AVAILABLE, the command 'sql.select from OUser where name = ? limit 1' against database 'web' will be executed only on local server with the possibility to have partial result [ODistributedStorage] 2020-06-17 09:58:20:625 INFO [ograph01-5-71] Reassigning ownership of clusters for database web... [OHazelcastPlugin] 2020-06-17 09:58:20:625 INFO [ograph01-5-71] Reassignment of clusters for database 'web' completed (classes=167) [OHazelcastPlugin] 2020-06-17 09:58:20:631 INFO Shutting down plugins: [OServerPluginManager] 2020-06-17 09:58:20:631 INFO - studio [OServerPluginManager] 2020-06-17 09:58:20:631 INFO - cluster [OServerPluginManager] 2020-06-17 09:58:20:631 WARNI Shutting down node 'ograph01-5-71'... [OHazelcastPlugin] 2020-06-17 09:58:20:631 INFO [ograph01-5-71] Updated node status to 'SHUTTINGDOWN' [OHazelcastPlugin] 2020-06-17 09:58:20:647 INFO [ograph01-5-71] Received updated status ograph01-5-71.web=NOT_AVAILABLE [OHazelcastPlugin] 2020-06-17 09:58:20:647 INFO [ograph01-5-71] Distributed servers status (*=current @=lockmgr[ograph01-5-71]): +-------------------+------+--------------------------+-----+---------+----------------+----------------+----------------------+ |Name |Status|Databases |Conns|StartedOn|Binary |HTTP |UsedMemory | +-------------------+------+--------------------------+-----+---------+----------------+----------------+----------------------+ |ograph01-5-71(*)(@)|ONLINE|web=NOT_AVAILABLE (MASTER)|31 |09:51:19 |172.16.5.71:2424|172.16.5.71:2480|1.45GB/64.00GB (2.27%)| +-------------------+------+--------------------------+-----+---------+----------------+----------------+----------------------+ [OHazelcastPlugin] 2020-06-17 09:58:21:597 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] All migration tasks have been completed, queues are empty. [MigrationThread] 2020-06-17 09:58:21:621 INFO [ograph01-5-71] Shutting down distributed database manager 'web'. Pending objects: txs=0 locks=0 [ODistributedDatabaseImpl] 2020-06-17 09:58:21:623 INFO [ograph01-5-71] Updated node status to 'OFFLINE' [OHazelcastPlugin] 2020-06-17 09:58:21:623 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] [172.16.5.71]:2434 is SHUTTING_DOWN [LifecycleService] 2020-06-17 09:58:21:623 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Shutdown request of [172.16.5.71]:2434 is handled [MigrationManager] 2020-06-17 09:58:21:851 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Re-partitioning cluster data... Migration queue size: 188 [MigrationManager] 2020-06-17 09:58:22:513 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Shutting down connection manager... [Node] 2020-06-17 09:58:22:514 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Connection[id=2, /172.16.5.71:2434->/172.16.5.73:60981, endpoint=[172.16.5.73]:2434, alive=false, type=MEMBER] closed. Reason: TcpIpConnectionManager is stopping [TcpIpConnection] 2020-06-17 09:58:22:514 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Connection[id=1, /172.16.5.71:2434->/172.16.5.72:50517, endpoint=[172.16.5.72]:2434, alive=false, type=MEMBER] closed. Reason: TcpIpConnectionManager is stopping [TcpIpConnection] 2020-06-17 09:58:22:515 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Shutting down node engine... [Node] 2020-06-17 09:58:23:845 INFO [ograph01-5-71] Removing server 'ograph04-5-74' from all the databases (removeOnlyDynamicServers=true)... [OHazelcastPlugin] 2020-06-17 09:58:25:524 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Destroying node NodeExtension. [NodeExtension] 2020-06-17 09:58:25:524 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] Hazelcast Shutdown is completed in 3901 ms. [Node] 2020-06-17 09:58:25:524 INFO [172.16.5.71]:2434 [orientdb] [3.10.6] [172.16.5.71]:2434 is SHUTDOWN [LifecycleService] 2020-06-17 09:58:25:525 INFO - teleporter [OServerPluginManager] 2020-06-17 09:58:25:525 INFO - custom-sql-functions-manager [OServerPluginManager] 2020-06-17 09:58:25:525 INFO - script-interpreter [OServerPluginManager] 2020-06-17 09:58:25:526 INFO - etl [OServerPluginManager] 2020-06-17 09:58:25:526 INFO - neo4j-importer [OServerPluginManager] 2020-06-17 09:58:25:526 INFO Shutting down databases: [OServer] 2020-06-17 09:58:25:526 INFO Orient Engine is shutting down... [Orient] 2020-06-17 09:58:25:529 INFO - shutdown storage: OSystem... [OrientDBDistributed] 2020-06-17 09:58:25:595 INFO - shutdown storage: web... [OrientDBDistributed] 2020-06-17 09:58:25:979 INFO Clearing byte buffer pool [Orient] 2020-06-17 09:58:26:291 INFO OrientDB Engine shutdown complete [Orient] 2020-06-17 09:58:26:291 INFO OrientDB Server shutdown complete [OServer]