Optimize the write performance when host is down

[luyade]

What type of PR is this?

• enhancement

What problem(s) does this PR solve?

Issue(s) number: #5570

Description:

Optimize the write performance after some host is down.

How do you solve it?

Special notes for your reviewer, ex. impact of this fix, design document, etc:

Checklist:

Tests:

• Unit test(positive and negative cases)
• Function test
• Performance test
• N/A

Affects:

• Documentation affected (Please add the label if documentation needs to be modified.)
• Incompatibility (If it breaks the compatibility, please describe it and add the label.）
• If it's needed to cherry-pick (If cherry-pick to some branches is required, please label the destination version(s).)
• Performance impacted: Consumes more CPU/Memory

Release notes:

Please confirm whether to be reflected in release notes and how to describe:

ex. Fixed the bug .....

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

Attention: Patch coverage is 46.42857% with 15 lines in your changes missing coverage. Please review.

Project coverage is 78.22%. Comparing base (e469b32) to head (ccaa956).
Report is 69 commits behind head on master.

Files with missing lines	Patch %	Lines
src/kvstore/raftex/Host.cpp	46.42%	15 Missing ⚠️

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #5571      +/-   ##
==========================================
+ Coverage   76.96%   78.22%   +1.26%     
==========================================
  Files        1130     1130              
  Lines       85345    85368      +23     
==========================================
+ Hits        65686    66783    +1097     
+ Misses      19659    18585    -1074     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[luyade]

@critical27 any additional thoughts?

[critical27]

If a host is down, the leader will still continue to send appendLog requests to the bad follower. As time goes on, the log gap will become large which will make the performance of write request bad.

The key points that make performance regression is that, when log gap become large, the building of the WAL iterator from specific logId is too slow. (In the same wal file, the target log is seek in sequential)

I'm ok with this PR for now. If you are interested in the problem I said, could try to fix it. The problem is the biggest one in current implementation. It will make raft much more robust and faster in many scenarios.

[critical27]

One point you need to check is that, when a node is elected as leader, should we reset all paused_ of its host to false. WDYT?

[luyade]

If a host is down, the leader will still continue to send appendLog requests to the bad follower. As time goes on, the log gap will become large which will make the performance of write request bad.

The key points that make performance regression is that, when log gap become large, the building of the WAL iterator from specific logId is too slow. (In the same wal file, the target log is seek in sequential)

I'm ok with this PR for now. If you are interested in the problem I said, could try to fix it. The problem is the biggest one in current implementation. It will make raft much more robust and faster in many scenarios.

Actually I fixed the issue you mentioned (at least from my view) in our implementation and it almost had no peformance regression even if the log gap is very large. ^_^

[luyade]

One point you need to check is that, when a node is elected as leader, should we reset all paused_ of its host to false. WDYT?

Yes, the current implementation will call host->resume to reset paused_ as false. I think it should be fine. At the beginning, the new leader doesn't know its peer is down. It can eventually set the right status of the peer by sending HB request.