Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Constraints to de-prioritize nodes from becoming shard allocation targets #489

Closed
wants to merge 1 commit into from
Closed

Constraints to de-prioritize nodes from becoming shard allocation targets #489

wants to merge 1 commit into from

Conversation

ashwinpankaj
Copy link
Contributor

Signed-off-by: Ashwin Pankaj appankaj@amazon.com

Description

An allocation constraint mechanism, that de-prioritizes nodes from getting picked for allocation if they breach certain constraints

Issues Resolved

#487

Check List

  • [x ] New functionality includes testing.
    • [x ] All tests pass
  • [x ] New functionality has been documented.
    • [x ] New functionality has javadoc added
  • [x ] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Constraints to de-prioritize nodes from becoming shard allocation tar…
647f72a 
…gets

Signed-off-by: Ashwin Pankaj <appankaj@amazon.com>
@odfe-release-bot
Copy link

✅   Gradle Wrapper Validation success 647f72a

@odfe-release-bot
Copy link

✅   DCO Check Passed 647f72a

@odfe-release-bot
Copy link

❌   Gradle Precommit failure 647f72a
Log 79

@peterzhuamazon
Copy link
Member

start gradle precommit
start dco check
start wrapper validation

@odfe-release-bot
Copy link

✅   DCO Check Passed 647f72a

@odfe-release-bot
Copy link

✅   Gradle Wrapper Validation success 647f72a

@peterzhuamazon
Copy link
Member

start gradle precommit

@odfe-release-bot
Copy link

❌   Gradle Precommit failure 647f72a
Log 249

shwetathareja
shwetathareja reviewed Apr 18, 2021
View reviewed changes
Copy link
Member

@shwetathareja shwetathareja left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

did the first pass, pending tests review

server/src/main/java/org/opensearch/cluster/routing/allocation/AllocationConstraints.java Show resolved Hide resolved
...c/main/java/org/opensearch/cluster/routing/allocation/allocator/BalancedShardsAllocator.java Show resolved Hide resolved
...c/main/java/org/opensearch/cluster/routing/allocation/allocator/BalancedShardsAllocator.java
@@ -204,14 +234,15 @@ public float getShardBalance() {
* </ul>
* <code>weight(node, index) = weight<sub>index</sub>(node, index) + weight<sub>node</sub>(node, index)</code>
*/
private static class WeightFunction {
public static class WeightFunction {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lets keep the class private along with its constructor or any method which has been changed to public

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this was needed for tests - so we can have more scoped UTs that test the weight fn alone. Existing balancer tests always tested the full reroute path. We can explore if package private helps.

itiyamas
itiyamas suggested changes Apr 20, 2021
View reviewed changes
Copy link
Contributor

@itiyamas itiyamas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have not reviewed the tests yet. It will be tricky to explain the new weight function now, especially with more constraints.

server/src/main/java/org/opensearch/cluster/routing/allocation/AllocationConstraints.java Show resolved Hide resolved
server/src/main/java/org/opensearch/cluster/routing/allocation/AllocationConstraints.java Show resolved Hide resolved
server/src/main/java/org/opensearch/cluster/routing/allocation/AllocationConstraints.java Show resolved Hide resolved
...c/main/java/org/opensearch/cluster/routing/allocation/allocator/BalancedShardsAllocator.java Show resolved Hide resolved
...c/main/java/org/opensearch/cluster/routing/allocation/allocator/BalancedShardsAllocator.java Show resolved Hide resolved
server/src/main/java/org/opensearch/cluster/routing/allocation/AllocationConstraints.java Show resolved Hide resolved
vigyasharma
vigyasharma reviewed Apr 28, 2021
View reviewed changes
Copy link
Contributor

@vigyasharma vigyasharma left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for merging this, @ashwinpankaj ! Good to see this make it upstream.

server/src/main/java/org/opensearch/cluster/routing/allocation/AllocationConstraints.java Show resolved Hide resolved
server/src/main/java/org/opensearch/cluster/routing/allocation/AllocationConstraints.java Show resolved Hide resolved
server/src/main/java/org/opensearch/cluster/routing/allocation/AllocationConstraints.java Show resolved Hide resolved
...c/main/java/org/opensearch/cluster/routing/allocation/allocator/BalancedShardsAllocator.java Show resolved Hide resolved
...c/main/java/org/opensearch/cluster/routing/allocation/allocator/BalancedShardsAllocator.java
@@ -204,14 +234,15 @@ public float getShardBalance() {
* </ul>
* <code>weight(node, index) = weight<sub>index</sub>(node, index) + weight<sub>node</sub>(node, index)</code>
*/
private static class WeightFunction {
public static class WeightFunction {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this was needed for tests - so we can have more scoped UTs that test the weight fn alone. Existing balancer tests always tested the full reroute path. We can explore if package private helps.

server/src/main/java/org/opensearch/cluster/routing/allocation/AllocationConstraints.java Show resolved Hide resolved
server/src/main/java/org/opensearch/cluster/routing/allocation/AllocationConstraints.java
}

/**
* {@link ConstraintMode} values define possible modes in which allocation constraints can be enabled on
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can remove the ConstraintMode parameter, and always use it for unassigned shards only. That is, enabled would apply this to allocate unassigned operation, and disabled will turn this off.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@ashwinpankaj ashwinpankaj mentioned this pull request May 10, 2021
Merged
5 tasks
@setiah
Copy link
Contributor

setiah commented May 25, 2021

@ashwinpankaj can we close this, since we have #680 ?

@ashwinpankaj
Copy link
Contributor Author

closed in favour of #680

ritty27 pushed a commit to ritty27/OpenSearch that referenced this pull request May 12, 2024
@szczepanczykd
fix: change int type to long (opensearch-project#489)
95969db 
* fix: change int type to long

Signed-off-by: Dominik Szczepanczyk <szczepanczyk.dominik@gmail.com>

* update CHANGELOG.md

Signed-off-by: Dominik Szczepanczyk <szczepanczyk.dominik@gmail.com>

* fix: change totalDataSetSizeInBytes to Long too

Signed-off-by: Dominik Szczepanczyk <szczepanczyk.dominik@gmail.com>

* fix: skip removing opensearch-observability index in AfterTest cleanup

Signed-off-by: Dominik Szczepanczyk <szczepanczyk.dominik@gmail.com>

---------

Signed-off-by: Dominik Szczepanczyk <szczepanczyk.dominik@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@vigyasharma vigyasharma vigyasharma left review comments

@shwetathareja shwetathareja shwetathareja left review comments

@itiyamas itiyamas itiyamas requested changes

@anasalkouz anasalkouz Awaiting requested review from anasalkouz anasalkouz will be requested when the pull request is marked ready for review anasalkouz is a code owner

@andrross andrross Awaiting requested review from andrross andrross will be requested when the pull request is marked ready for review andrross is a code owner

@ashking94 ashking94 Awaiting requested review from ashking94 ashking94 will be requested when the pull request is marked ready for review ashking94 is a code owner

@Bukhtawar Bukhtawar Awaiting requested review from Bukhtawar Bukhtawar will be requested when the pull request is marked ready for review Bukhtawar is a code owner

@CEHENKLE CEHENKLE Awaiting requested review from CEHENKLE CEHENKLE will be requested when the pull request is marked ready for review CEHENKLE is a code owner

@cwperks cwperks Awaiting requested review from cwperks cwperks will be requested when the pull request is marked ready for review cwperks is a code owner

@dblock dblock Awaiting requested review from dblock dblock will be requested when the pull request is marked ready for review dblock is a code owner

@dbwiddis dbwiddis Awaiting requested review from dbwiddis dbwiddis will be requested when the pull request is marked ready for review dbwiddis is a code owner

@gbbafna gbbafna Awaiting requested review from gbbafna gbbafna will be requested when the pull request is marked ready for review gbbafna is a code owner

@jainankitk jainankitk Awaiting requested review from jainankitk jainankitk will be requested when the pull request is marked ready for review jainankitk is a code owner

@kotwanikunal kotwanikunal Awaiting requested review from kotwanikunal kotwanikunal will be requested when the pull request is marked ready for review kotwanikunal is a code owner

@linuxpi linuxpi Awaiting requested review from linuxpi linuxpi will be requested when the pull request is marked ready for review linuxpi is a code owner

@mch2 mch2 Awaiting requested review from mch2 mch2 will be requested when the pull request is marked ready for review mch2 is a code owner

@msfroh msfroh Awaiting requested review from msfroh msfroh will be requested when the pull request is marked ready for review msfroh is a code owner

@nknize nknize Awaiting requested review from nknize nknize will be requested when the pull request is marked ready for review nknize is a code owner

@owaiskazi19 owaiskazi19 Awaiting requested review from owaiskazi19 owaiskazi19 will be requested when the pull request is marked ready for review owaiskazi19 is a code owner

@reta reta Awaiting requested review from reta reta will be requested when the pull request is marked ready for review reta is a code owner

@Rishikesh1159 Rishikesh1159 Awaiting requested review from Rishikesh1159 Rishikesh1159 will be requested when the pull request is marked ready for review Rishikesh1159 is a code owner

@sachinpkale sachinpkale Awaiting requested review from sachinpkale sachinpkale will be requested when the pull request is marked ready for review sachinpkale is a code owner

@saratvemulapalli saratvemulapalli Awaiting requested review from saratvemulapalli saratvemulapalli will be requested when the pull request is marked ready for review saratvemulapalli is a code owner

@sohami sohami Awaiting requested review from sohami sohami will be requested when the pull request is marked ready for review sohami is a code owner

@VachaShah VachaShah Awaiting requested review from VachaShah VachaShah will be requested when the pull request is marked ready for review VachaShah is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@ashwinpankaj @odfe-release-bot @peterzhuamazon @setiah @vigyasharma @shwetathareja @itiyamas