Add method (*EtcdServer) IsRaftLoopBlocked to support checking whether the raft loop is blocked

[ahrtr]

Link to #16007

Please read https://github.com/etcd-io/etcd/blob/main/CONTRIBUTING.md#contribution-flow.

[ahrtr]

cc @serathius

[chaochn47]

Copied from the design doc comment:

A counter could be added to the raftNode.tick function [1] and the prober could just look up the counter to decide if the raft loop is deadlocked or not.

[1] https://github.com/etcd-io/etcd/blob/aa97484166d2b3fb6afeb4390344e68b02afb566/server/etcdserver/raft.go#L155-L159

Since the raft loop deadlock will block the next select statement execution:

I can see there are two approaches:

1. prober sends a request to the etcd server and waits for a response back with a configurable waiting timeout.
2. prober queries the etcd server if in the past x second, there is at least one select statement execution which is the raft tick timer. It immediately sends the response back.

With the goal of prober check fittng in the 1s timeout, looks like the 2nd approach is better, what do you think? @ahrtr

[chaochn47]

cc @siyuanfoundation ^

[ahrtr]

2. prober queries the etcd server if in the past x second, there is at least one select statement execution which is the raft tick timer. It immediately sends the response back.

This means that you need to remember the timestamp of the last tick, and check it in the liveness probe something like time.Sub(now - t) > timeout. You will get immediate result, but It will be affected when clock drift. I would suggest to avoid this approach.

Note that this PR just provides a basic functionality for checking if the raftloop blocks. You can check it async.

[chaochn47]

This means that you need to remember the timestamp of the last tick, and check it in the liveness probe something like time.Sub(now - t) > timeout. You will get immediate result, but It will be affected when clock drift. I would suggest to avoid this approach.

It's not necessary, sent a drafted PR to demonstrate #16713.

Note that this PR just provides a basic functionality for checking if the raftloop blocks. You can check it async.

Yeah, assuming the "async" here means try send to the dummy channel and then go ahead to the remaining checks in the prober and then validate the previous sent has completed. It seems complicated compared with the counter approach. WDYT?

[siyuanfoundation]

Copied from the design doc comment:

A counter could be added to the raftNode.tick function [1] and the prober could just look up the counter to decide if the raft loop is deadlocked or not.

[1] https://github.com/etcd-io/etcd/blob/aa97484166d2b3fb6afeb4390344e68b02afb566/server/etcdserver/raft.go#L155-L159

Since the raft loop deadlock will block the next select statement execution:

I can see there are two approaches:

1. prober sends a request to the etcd server and waits for a response back with a configurable waiting timeout.
2. prober queries the etcd server if in the past x second, there is at least one select statement execution which is the raft tick timer. It immediately sends the response back.

With the goal of prober check fittng in the 1s timeout, looks like the 2nd approach is better, what do you think? @ahrtr

Copied from the design doc comment:

A counter could be added to the raftNode.tick function [1] and the prober could just look up the counter to decide if the raft loop is deadlocked or not.

[1] https://github.com/etcd-io/etcd/blob/aa97484166d2b3fb6afeb4390344e68b02afb566/server/etcdserver/raft.go#L155-L159

Since the raft loop deadlock will block the next select statement execution:

I can see there are two approaches:

1. prober sends a request to the etcd server and waits for a response back with a configurable waiting timeout.
2. prober queries the etcd server if in the past x second, there is at least one select statement execution which is the raft tick timer. It immediately sends the response back.

With the goal of prober check fittng in the 1s timeout, looks like the 2nd approach is better, what do you think? @ahrtr

I think either way, the interval between checks could be too short to differentiate a slow ready process from a deadlock. I suggest using another longer ticker to reset the count instead of resetting based on probes in #16713

[chaochn47]

I think either way, the interval between checks could be too short to differentiate a slow ready process from a deadlock.

In the second approach, it's determined by the administrator case by case. 'prober interval * failure threshold' should be a sane value based on administrator judgement. e.g. if they are using a network based volume like EBS or physically attached SSD..

I suggest using another longer ticker to reset the count instead of resetting based on probes in #16713

Adding another ticker may not be optimal, how will you plan set the new ticker interval, is it configurable? It may make etcd set up more complicated.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.