Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Repository] Honor throttling limits on recovery for multistream downloads #10282

Closed
Tracked by #8596
andrross opened this issue Sep 29, 2023 · 1 comment
Closed
Tracked by #8596
Assignees
Labels
distributed framework enhancement Enhancement or improvement to existing feature or request

Comments

@andrross
Copy link
Member

The relevant settings here are:

Field Data type Description
indices.recovery.max_bytes_per_sec String Limits the total inbound and outbound recovery traffic for each node. This applies to peer recoveries and snapshot recoveries. Default is 40mb. If you set the recovery traffic value to less than or equal to 0mb, rate limiting will be disabled, which causes recovery data to be transferred at the highest possible rate.
indices.recovery.max_concurrent_file_chunks Integer The number of file chunks sent in parallel for each recovery operation. Default is 2.
indices.recovery.max_concurrent_operations Integer The number of operations sent in parallel for each recovery. Default is 1.

I think max_bytes_per_sec should clearly be honored. Implementation should be a matter of wrapping the input streams with the rate limiting stream functionality that already exists.

max_concurrent_file_chunks and max_concurrent_operations are less clear. I think they were intended for limiting the peer-to-peer recovery. If the intent is to limit the load on the source node, then perhaps it is not relevant when the source is a remote object store.

@andrross andrross added bug Something isn't working untriaged labels Sep 29, 2023
@andrross andrross added distributed framework enhancement Enhancement or improvement to existing feature or request and removed bug Something isn't working untriaged labels Sep 29, 2023
@andrross andrross self-assigned this Oct 2, 2023
@kotwanikunal
Copy link
Member

Resolved by #10349

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
distributed framework enhancement Enhancement or improvement to existing feature or request
Projects
None yet
Development

No branches or pull requests

3 participants
@andrross @kotwanikunal and others