Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Third solution] Fix the potential data loss for clusters with only one member (raft layer change) #14407

Closed
wants to merge 1 commit into from

Conversation

ahrtr
Copy link
Member

@ahrtr ahrtr commented Aug 31, 2022

Third solution to fix #14370

For a cluster with only one member, the raft always send identical
unstable entries and committed entries to etcdserver, and etcd
responds to the client once it finishes (actually partially) the
applying workflow.

When the client receives the response, it doesn't mean etcd has already
successfully saved the data, including BoltDB and WAL, because:

  1. etcd commits the boltDB transaction periodically instead of on each request;
  2. etcd saves WAL entries in parallel with applying the committed entries.
    Accordingly, it may run into a situation of data loss when the etcd crashes
    immediately after responding to the client and before the boltDB and WAL
    successfully save the data to disk.
    Note that this issue can only happen for clusters with only one member.

For clusters with multiple members, it isn't an issue, because etcd will
not commit & apply the data before it being replicated to majority members.
When the client receives the response, it means the data must have been applied.
It further means the data must have been committed.
Note: for clusters with multiple members, the raft will never send identical
unstable entries and committed entries to etcdserver.

Signed-off-by: Benjamin Wang wachao@vmware.com

Please read https://github.com/etcd-io/etcd/blob/main/CONTRIBUTING.md#contribution-flow.

@ahrtr ahrtr changed the title [Third solution] Fix the potential data loss for clusters with only one member [Third solution] Fix the potential data loss for clusters with only one member (raft layer change) Aug 31, 2022
@ahrtr ahrtr force-pushed the one_member_data_loss_raft_protocol branch from 7a53bab to 97e54b2 Compare August 31, 2022 01:33
For a cluster with only one member, the raft always send identical
unstable entries and committed entries to etcdserver, and etcd
responds to the client once it finishes (actually partially) the
applying workflow.

When the client receives the response, it doesn't mean etcd has already
successfully saved the data, including BoltDB and WAL, because:
   1. etcd commits the boltDB transaction periodically instead of on each request;
   2. etcd saves WAL entries in parallel with applying the committed entries.
Accordingly, it may run into a situation of data loss when the etcd crashes
immediately after responding to the client and before the boltDB and WAL
successfully save the data to disk.
Note that this issue can only happen for clusters with only one member.

For clusters with multiple members, it isn't an issue, because etcd will
not commit & apply the data before it being replicated to majority members.
When the client receives the response, it means the data must have been applied.
It further means the data must have been committed.
Note: for clusters with multiple members, the raft will never send identical
unstable entries and committed entries to etcdserver.

Signed-off-by: Benjamin Wang <wachao@vmware.com>
@ahrtr ahrtr force-pushed the one_member_data_loss_raft_protocol branch from 97e54b2 to 87f0a76 Compare August 31, 2022 01:39
@ahrtr
Copy link
Member Author

ahrtr commented Sep 5, 2022

Closing this PR as we agreed that we will not proceed with this PR.

@ahrtr ahrtr closed this Sep 5, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

Durability API guarantee broken in single node cluster
1 participant