memberlist gossip

[stuartnelson3]

meant to address #1387

one current issue is that it seems peer0 has a few notifications queued, but hasn't cleared them in over an hour. Some way of clearing old messages is probably needed.

The method for choosing RetransmitMulti is also something to be discussed. It's an important value, but I'm wary of exposing too many configuration options in alertmanager to alter memberlist. I think we need to come up with a "smart" value that also doesn't overwhelm the transmit queues .

[simonpasquier]

Do we want to re-gossip new/updated silences too?

[simonpasquier]

nit but I would do the computation here rather than add a new argument to newDelegate().

[stuartnelson3]

you mean setting RetransmitMulti on p.delegate?

[simonpasquier]

Sorry for being inaccurate. I meant:

retransmit := len(knownPeers)/2
if restransmit < 3 {
    retransmit = 3
}
p.delegate = newDelegate(l, reg, p, retransmit)

[stuartnelson3]

Do we want to re-gossip new/updated silences too?

Yeah, that would make sense

[stuartnelson3]

Updated.

When running this branch at SC with only nflog entries being further gossiped, we were seeing messages queued up. Newer messages are sent preferentially by memberlist, but it's still an issue that we have an unbounded queue. Ideally, the queue would be able to empty itself, but that's not what I was seeing. I'll investigate this.

[mxinden]

The method for choosing RetransmitMulti is also something to be discussed. It's an important value, but I'm wary of exposing too many configuration options in alertmanager to alter memberlist. I think we need to come up with a "smart" value that also doesn't overwhelm the transmit queues .

I agree with keeping the amount of configuration options small (fow now).

What is your reasoning for:

retransmit := len(knownPeers) / 2

Thanks a lot for working on this!

[stuartnelson3]

Note: The probability calculation I was doing was based on changing GossipPeers, not RetransmitMult. Both are important values, and I tried to factor them both in for this.

• RetransmitMult is how many times a message will be sent. Each send goes to one machine.
• GossipPeers is how many machines to gossip to in a single gossip interval.

Each gossip interval will send messages to all machines picked by kRandomNodes, but it will not necessarily exhaust all retransmits of a message; the message may remain in the queue to be sent at the next gossip interval. But, if numNodes > retransmits, then a message will only be sent to the first N nodes, where N = retransmits. I imagine this is why I was seeing messages being queued -- I was instructing memberlist to send a message many times, but based on the number of peers it was sending messages to, it wouldn't hit that retransmit limit and remove the message from its internal queue. Newer gossip messages would arrive, and the older ones could never be sent.

Probability of a machine not being gossiped to during a gossip interval

Caveat: This might be wrong :)

Play along at:
https://gist.github.com/stuartnelson3/81ba1f750196e9fccaf2f7b23bdbf5ac

Tweaking the numbers available (probabilities are the chance of not receiving a message, and given as a percentage out of 100):

retransmits	GossipPeers	num AMs	p(NoGossip)	p(Gossip)
1	3	3	50.0030	25.0030
2	3	3	2.6065	0.0679
3	3	3	2.6065	0.0679
3	3	4	3.1591	0.0032
3	3	10	66.6667	2.6012
5	5	10	44.4444	0.0677

p(Gossip) is if each node is configured to gossip only newly received messages. Any message they've already seen will not be gossiped.

The testing was far from thorough given that I doubt anyone runs more than 10 instances, but it seemed like len(knownPeers) / 2 was a good place to start. If more peers join, however, maybe this value should be updated.