Adding index create block when all nodes have breached high disk watermark #5852
+222
−23
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Gradle Check (Jenkins) Run Completed with:
|
fd39829 to
76b9782
Compare
Gradle Check (Jenkins) Run Completed with:
|
Gradle Check (Jenkins) Run Completed with:
|
Gradle Check (Jenkins) Run Completed with:
|
76b9782 to
dee1ea0
Compare
Gradle Check (Jenkins) Run Completed with:
|
dee1ea0 to
79bade6
Compare
Gradle Check (Jenkins) Run Completed with:
|
79bade6 to
4ee5693
Compare
Gradle Check (Jenkins) Run Completed with:
|
Codecov Report
@@ Coverage Diff @@
## main #5852 +/- ##
============================================
+ Coverage 70.57% 70.84% +0.27%
- Complexity 58486 58730 +244
============================================
Files 4770 4770
Lines 280721 280743 +22
Branches 40536 40539 +3
============================================
+ Hits 198123 198901 +778
+ Misses 66228 65502 -726
+ Partials 16370 16340 -30
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. |
a8787d6 to
888c720
Compare
Gradle Check (Jenkins) Run Completed with:
|
888c720 to
b5b12f3
Compare
Gradle Check (Jenkins) Run Completed with:
|
Gradle Check (Jenkins) Run Completed with:
|
dceeb82 to
b8dfc29
Compare
Gradle Check (Jenkins) Run Completed with:
|
b8dfc29 to
e6ac695
Compare
Gradle Check (Jenkins) Run Completed with:
|
6ba0f38 to
08ae9be
Compare
Gradle Check (Jenkins) Run Completed with:
|
gbbafna
reviewed
Jan 19, 2023
server/src/main/java/org/opensearch/cluster/routing/allocation/DiskThresholdMonitor.java
Show resolved
Hide resolved
…rmark Signed-off-by: Rishav Sagar <rissag@amazon.com>
b8d232d to
f401496
Compare
Gradle Check (Jenkins) Run Completed with:
|
Gradle Check (Jenkins) Run Completed with:
|
gbbafna
approved these changes
Jan 19, 2023
Arpit-Bandejiya
approved these changes
Jan 19, 2023
Bukhtawar
reviewed
Jan 19, 2023
Comment on lines
411
to
437
| ActionListener<Void> wrappedListener = ActionListener.wrap(r -> { | ||
| setLastRunTimeMillis(); | ||
| listener.onResponse(r); | ||
| }, e -> { | ||
| ActionListener<Void> wrappedListener = ActionListener.wrap(r -> { listener.onResponse(r); }, e -> { |
There was a problem hiding this comment.
for my understanding y did we remove last run time?
There was a problem hiding this comment.
As commented by Gaurav here, setLastRunTimeMillis() is only meant for reroute operations as it needs to honor diskThresholdSettings.getRerouteInterval().millis(), so removing it from updateIndicesReadOnly as well.
There was a problem hiding this comment.
If we remove setLastRunTimeMillis it will try to reroute once even though we don't need to reroute.
There was a problem hiding this comment.
Addressed the comment
Signed-off-by: Rishav Sagar <rissag@amazon.com>
f401496 to
0c5ed1f
Compare
Gradle Check (Jenkins) Run Completed with:
|
Bukhtawar
approved these changes
Jan 19, 2023
6 tasks
andrross
pushed a commit
that referenced
this pull request
Feb 2, 2023
…rmark (#5852) (#6165) * Adding index create block when all nodes have breached high disk watermark (cherry picked from commit 6309611) Signed-off-by: Rishav Sagar <rissag@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
|
@RS146BIJAY: Will the index creation cluster block be lifted automatically when one or more nodes fall below the watermark again? |
6 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
…rmark
Signed-off-by: Rishav Sagar rissag@amazon.com
Description
OpenSearch stops allocating any shards to nodes that have breached High Disk Watermark. If a scenario arises that all the nodes in the cluster breached high disk watermark, no new shards will be created on this cluster. Now if we try to create a new index on this cluster, it will be a red index (since no primary or replica shards can be created for this index).
In order to prevent red cluster, we propose that whenever all the nodes in the cluster has breached high disk watermark we apply a INDEX CREATE BLOCK on the cluster to prevent creation of any red indices. We will modify the DiskThresholdMonitor (which monitors for disk watermarks thresholds on a domain and take appropriate action) to include an extra check on whether low disk watermark is breached on all the nodes in cluster. If it has, it will apply an index create block on the entire cluster.
Issues Resolved
#4456
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.