node fails to join cluster after upgrade 6.54 -> 6.7.0 #40784
Comments
|
This sounds like the issue fixed by #40483. You can work around this by first upgrading the cluster completely to 6.6.2 and thence to 6.7.0. Alternatively you can wait for 6.7.1 to be released.
|
|
@DaveCTurner when 6.7.1 to be release? |
|
@DaveCTurner upgrading to 6.6.2 first didn't work. any other workarounds you know? thanks! |
|
@ItamarBenjamin check this way #40511 (comment) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
** this does not seem related to #40565
Elasticsearch version (
bin/elasticsearch --version): cluster is 6.5.4, upgraded single node to 6.7.0. all docker buildsPlugins installed: none
JVM version (
java -version): docker provided jvmOS version (
uname -aif on a Unix-like system): 4.15.0-33-generic #36~16.04.1-UbuntuDescription of the problem including expected versus actual behavior:
upgraded a single node on 5 different clusters, all clusters suffered the same issue. upgraded node was not able to rejoin the cluster and remained yellow. to fix had to upgrade the other nodes to 6.7.0, causing downtime.
Steps to reproduce:
run a 3 6.5.4 nodes cluster on docker, upgrade a single node to 6.7.0
Provide logs (if relevant):
[2019-04-03T06:30:56,549][WARN ][o.e.d.z.PublishClusterStateAction] [fmyinf7012] publishing cluster state with version [29318] failed for the following nodes: [[{fmyinf7011}{ejFPzwZHRxqk4CNBtF66iA}{x1NPhmVuQcucbPX3VeEgPg}{10.96.87.25}{10.96.87.25:9300}{xpack.installed=true}]] [2019-04-03T06:30:56,563][INFO ][o.e.c.s.MasterService ] [fmyinf7012] zen-disco-node-failed({fmyinf7011}{ejFPzwZHRxqk4CNBtF66iA}{x1NPhmVuQcucbPX3VeEgPg}{10.96.87.25}{10.96.87.25:9300}{xpack.installed=true}), reason(transport disconnected)[{fmyinf7011}{ejFPzwZHRxqk4CNBtF66iA}{x1NPhmVuQcucbPX3VeEgPg}{10.96.87.25}{10.96.87.25:9300}{xpack.installed=true} transport disconnected, {fmyinf7011}{ejFPzwZHRxqk4CNBtF66iA}{x1NPhmVuQcucbPX3VeEgPg}{10.96.87.25}{10.96.87.25:9300}{xpack.installed=true} transport disconnected], reason: removed {{fmyinf7011}{ejFPzwZHRxqk4CNBtF66iA}{x1NPhmVuQcucbPX3VeEgPg}{10.96.87.25}{10.96.87.25:9300}{xpack.installed=true},} [2019-04-03T06:30:56,569][INFO ][o.e.c.s.ClusterApplierService] [fmyinf7012] removed {{fmyinf7011}{ejFPzwZHRxqk4CNBtF66iA}{x1NPhmVuQcucbPX3VeEgPg}{10.96.87.25}{10.96.87.25:9300}{xpack.installed=true},}, reason: apply cluster state (from master [master {fmyinf7012}{K0Od0eOBTp6c8PRggjBpuw}{_1KFcV3zRryhiuGz7cCQwQ}{10.96.87.26}{10.96.87.26:9300}{xpack.installed=true} committed version [29319] source [zen-disco-node-failed({fmyinf7011}{ejFPzwZHRxqk4CNBtF66iA}{x1NPhmVuQcucbPX3VeEgPg}{10.96.87.25}{10.96.87.25:9300}{xpack.installed=true}), reason(transport disconnected)[{fmyinf7011}{ejFPzwZHRxqk4CNBtF66iA}{x1NPhmVuQcucbPX3VeEgPg}{10.96.87.25}{10.96.87.25:9300}{xpack.installed=true} transport disconnected, {fmyinf7011}{ejFPzwZHRxqk4CNBtF66iA}{x1NPhmVuQcucbPX3VeEgPg}{10.96.87.25}{10.96.87.25:9300}{xpack.installed=true} transport disconnected]]]) [2019-04-03T06:31:00,580][INFO ][o.e.c.s.MasterService ] [fmyinf7012] zen-disco-node-join[{fmyinf7011}{ejFPzwZHRxqk4CNBtF66iA}{x1NPhmVuQcucbPX3VeEgPg}{10.96.87.25}{10.96.87.25:9300}{xpack.installed=true}], reason: added {{fmyinf7011}{ejFPzwZHRxqk4CNBtF66iA}{x1NPhmVuQcucbPX3VeEgPg}{10.96.87.25}{10.96.87.25:9300}{xpack.installed=true},} [2019-04-03T06:31:00,635][INFO ][o.e.c.s.ClusterApplierService] [fmyinf7012] added {{fmyinf7011}{ejFPzwZHRxqk4CNBtF66iA}{x1NPhmVuQcucbPX3VeEgPg}{10.96.87.25}{10.96.87.25:9300}{xpack.installed=true},}, reason: apply cluster state (from master [master {fmyinf7012}{K0Od0eOBTp6c8PRggjBpuw}{_1KFcV3zRryhiuGz7cCQwQ}{10.96.87.26}{10.96.87.26:9300}{xpack.installed=true} committed version [29320] source [zen-disco-node-join[{fmyinf7011}{ejFPzwZHRxqk4CNBtF66iA}{x1NPhmVuQcucbPX3VeEgPg}{10.96.87.25}{10.96.87.25:9300}{xpack.installed=true}]]]) [2019-04-03T06:31:06,665][WARN ][o.e.t.n.Netty4Transport ] [fmyinf7012] exception caught on transport layer [NettyTcpChannel{localAddress=/10.96.87.26:58004, remoteAddress=10.96.87.25/10.96.87.25:9300}], closing connection java.lang.IllegalStateException: Message not fully read (response) for requestId [55772288], handler [org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler/org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1@58fac40a], error [false]; resetting at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1197) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:65) ~[transport-netty4-client-6.5.4.jar:6.5.4] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.30.Final.jar:4.1.30.Final] at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323) [netty-codec-4.1.30.Final.jar:4.1.30.Final] at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) [netty-codec-4.1.30.Final.jar:4.1.30.Final] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:426) [netty-codec-4.1.30.Final.jar:4.1.30.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) [netty-codec-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.30.Final.jar:4.1.30.Final] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241) [netty-handler-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) [netty-transport-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) [netty-transport-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) [netty-transport-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) [netty-transport-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) [netty-transport-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) [netty-transport-4.1.30.Final.jar:4.1.30.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) [netty-common-4.1.30.Final.jar:4.1.30.Final] at java.lang.Thread.run(Thread.java:834) [?:?]The text was updated successfully, but these errors were encountered: