Fixes for a failover scenario. #2260
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
Timeouts on client side should node affect inter-node communication. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
The problem is that accidental timeout errors can make us to ignore other nodes for some time. The primary purpose of the whole ignore mechanism is not to degrade in case of failover. For this case, closing connection and limiting the amount of dials is enough. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
Speed up big object construction in some cases. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
Currently, under a mixed load one failed PUT can lead to closing connection for all concurrent GETs. For PUT it does no harm: we have many other nodes to choose from. For GET we are limited by `REP N` factor, so in case of failover we can close the connection with the only node posessing an object, which leads to failing the whole operation. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
Codecov Report
@@ Coverage Diff @@
## support/v0.35 #2260 +/- ##
=================================================
- Coverage 31.01% 31.00% -0.01%
=================================================
Files 384 384
Lines 28453 28461 +8
=================================================
Hits 8824 8824
- Misses 18889 18897 +8
Partials 740 740
📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
acid-ant
approved these changes
Feb 17, 2023
carpawell
approved these changes
Feb 17, 2023
| @@ -128,7 +128,7 @@ func (exec execCtx) key() (*ecdsa.PrivateKey, error) { | |||
| } | |||
|
|
|||
| func (exec *execCtx) canAssemble() bool { | |||
| return exec.svc.assembly && !exec.isRaw() && !exec.headOnly() | |||
| return exec.svc.assembly && !exec.isRaw() && !exec.headOnly() && !exec.isLocal() | |||
There was a problem hiding this comment.
just to highlight: we have exec.svc.assembly and it is somehow always true
|
CHANGELOG would be also appreciated. |
|
Yes, we temporarily disabled compression. Currently, we cannot control disk load for policer (well, we can, but it is suboptimal) and it iterates over each shard in order. Failover case makes things worse because with high probability we will also read full object in memory to replicate. |
cthulhu-rider
added a commit
that referenced
this pull request
Feb 27, 2024
This reverts commit 907f427. GET requests with TTL=1 are absolutely correct regardless of the need to assemble the object. This TTL blocks request forwarding - it does not happen. At the same time, assembling an object on the server spawns new requests from scratch, which is not forwarding. The original commit does not have a description of the cause, and it was never discovered. Fixes #2447. Signed-off-by: Leonard Lyubich <leonard@morphbits.io>
cthulhu-rider
added a commit
that referenced
this pull request
Feb 27, 2024
This reverts commit 907f427. GET requests with TTL=1 are absolutely correct regardless of the need to assemble the object. This TTL blocks request forwarding - it does not happen. At the same time, assembling an object on the server spawns new requests from scratch, which is not forwarding. The original commit does not have a description of the cause, and it was never discovered. Fixes #2447. Signed-off-by: Leonard Lyubich <leonard@morphbits.io>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In master:
network/cache-related changes will be replaced with gRPC-native keepalive mechanism.Client cache separation should be approached more rigourously. It seems like a good idea, but we need to evaluate it further.