Nodes try to rejoin cluster when first listed node is down

[ccrebolder]

Regarding the line https://github.com/jjasghar/rabbitmq/blob/master/providers/cluster.rb#L202, it looks as if the elsif statement is checking whether var_node_name_to_join is part of cluster_status, but I think it should be checking var_node_name. var_node_name_to_join is just set to the first node name in the array passed into the lwrp.

I discovered that when powering down or stopping the first listed node in node['rabbitmq']['clustering']['cluster_nodes'], the other nodes worked fine until chef-client ran. They would then attempt to rejoin the cluster because the first node was no longer listed in the "running_nodes" output of rabbitmqctl cluster_status, and to rejoin they would try to connect again to the first node, which would fail as it was turned off. This would result in the whole cluster coming down.

Let me know if you'd like more info, or a PR for this.

[Rarian]

+1

[opsline-radek]

This fix breaks new cluster builds. When a new node comes up with cluster, it's alone part of the cluster itself. The check will always return true, it will never allow a node to join another one.

When I run cluster status on a new node I get this:

# rabbitmqctl cluster_status
Cluster status of node 'rabbit@production-rabbitmq-6' ...
[{nodes,[{disc,['rabbit@production-rabbitmq-6']}]},
 {running_nodes,['rabbit@production-rabbitmq-6']},
 {cluster_name,<<"mycluster">>},
 {partitions,[]},
 {alarms,[{'rabbit@production-rabbitmq-6',[]}]}]

Let's say, the cluster_nodes list contains production-rabbitmq-5 and production-rabbitmq-6. With the new code, node 6 will check whether itself is part of the cluster and according to output above, it is. It will never join 5. It should check whether node 5 is part of the cluster, then join it. The original code was correct.

The fix is simple - if the node is down and removed from chef server, update the cluster_nodes attributes to remove it - or better - make it dynamic based on a search in a wrapper cookbook.