fix cluster not able to spin up issue when disk usage exceeds threshold #15258

zane-neo · 2024-08-15T03:24:48Z

Description

root cause:

Observability plugin creates system index during cluster startup: link, and if cluster has index block state persisted, during the node startup this block will be applied to cluster state and index creation fails, then this exception is been thrown.
OpenSearch starts cluster in this method and the error will be thrown again util the OpenSearch startup command execution finished.
At the moment of the startup command complete, the JVM only has daemon threads without any non-daemon threads, and thus JVM exits, then cluster been shut down.

Changing the code of cluster start part to first start the keepAliveThread which is a non-daemon thread to make sure at least one non-daemon thread is running thus the JVM won't exit.

Related Issues

#14791

Check List

Functionality includes testing.
API changes companion pull request created, if applicable.
Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2024-08-15T03:35:55Z

❌ Gradle check result for d9096b2: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

dblock · 2024-08-15T19:00:19Z

Thanks for opening this! Would like someone who understands the code, maybe @andrross or @mch2 to take a look please?

In the meantime @zane-neo you should get this PR to green, with a CHANGELOG, etc.

zane-neo · 2024-08-16T01:51:23Z

Thanks for opening this! Would like someone who understands the code, maybe @andrross or @mch2 to take a look please?

In the meantime @zane-neo you should get this PR to green, with a CHANGELOG, etc.

Sure, this is a draft PR to prove this can fix the issue, but this fix has drawbacks, e.g. it changes the ClusterBlocks.java's field modifier, I'll figure out a better approach to fix this and make the checks green.

github-actions · 2024-08-16T06:40:03Z

❌ Gradle check result for 7ce9886: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-08-16T07:52:45Z

❌ Gradle check result for 3e8df68: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: zane-neo <zaniu@amazon.com>

github-actions · 2024-10-16T06:27:26Z

❌ Gradle check result for af9d70d: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

…ncy issue caused test failure Signed-off-by: zane-neo <zaniu@amazon.com>

github-actions · 2024-10-16T08:10:23Z

❌ Gradle check result for 7f24452: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-10-16T11:51:03Z

✅ Gradle check result for 7f24452: SUCCESS

dblock

I'm going to merge this as it has been sitting here for a while and is a visible bug fix, but @andrross do let @zane-neo know if you want more changes on top of it.

…ld (#15258) * fix cluster not able to spin up issue when disk usage exceeds threshold Signed-off-by: zane-neo <zaniu@amazon.com> * Add comment to changes Signed-off-by: zane-neo <zaniu@amazon.com> * Add UT to ensure the keepAliveThread starts before node starts Signed-off-by: zane-neo <zaniu@amazon.com> * remove unused imports Signed-off-by: zane-neo <zaniu@amazon.com> * Fix forbidden API calls check failed issue Signed-off-by: zane-neo <zaniu@amazon.com> * format code Signed-off-by: zane-neo <zaniu@amazon.com> * format code Signed-off-by: zane-neo <zaniu@amazon.com> * change setInstance method to static Signed-off-by: zane-neo <zaniu@amazon.com> * Add countdownlatch in test to coordinate the thread to avoid concureency issue caused test failure Signed-off-by: zane-neo <zaniu@amazon.com> --------- Signed-off-by: zane-neo <zaniu@amazon.com> (cherry picked from commit 62081f2) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…ld (opensearch-project#15258) * fix cluster not able to spin up issue when disk usage exceeds threshold Signed-off-by: zane-neo <zaniu@amazon.com> * Add comment to changes Signed-off-by: zane-neo <zaniu@amazon.com> * Add UT to ensure the keepAliveThread starts before node starts Signed-off-by: zane-neo <zaniu@amazon.com> * remove unused imports Signed-off-by: zane-neo <zaniu@amazon.com> * Fix forbidden API calls check failed issue Signed-off-by: zane-neo <zaniu@amazon.com> * format code Signed-off-by: zane-neo <zaniu@amazon.com> * format code Signed-off-by: zane-neo <zaniu@amazon.com> * change setInstance method to static Signed-off-by: zane-neo <zaniu@amazon.com> * Add countdownlatch in test to coordinate the thread to avoid concureency issue caused test failure Signed-off-by: zane-neo <zaniu@amazon.com> --------- Signed-off-by: zane-neo <zaniu@amazon.com>

… threshold (opensearch-project#15258)" This reverts commit 62081f2.

…x the issue. Signed-off-by: zane-neo <zaniu@amazon.com>

Signed-off-by: zane-neo <zaniu@amazon.com>

dblock · 2024-10-18T13:14:44Z

Thanks @zane-neo for hanging in here with us and @dbwiddis and @andrross for your thoughtful comments and help with moving forward!

…ld (opensearch-project#15258) * fix cluster not able to spin up issue when disk usage exceeds threshold Signed-off-by: zane-neo <zaniu@amazon.com> * Add comment to changes Signed-off-by: zane-neo <zaniu@amazon.com> * Add UT to ensure the keepAliveThread starts before node starts Signed-off-by: zane-neo <zaniu@amazon.com> * remove unused imports Signed-off-by: zane-neo <zaniu@amazon.com> * Fix forbidden API calls check failed issue Signed-off-by: zane-neo <zaniu@amazon.com> * format code Signed-off-by: zane-neo <zaniu@amazon.com> * format code Signed-off-by: zane-neo <zaniu@amazon.com> * change setInstance method to static Signed-off-by: zane-neo <zaniu@amazon.com> * Add countdownlatch in test to coordinate the thread to avoid concureency issue caused test failure Signed-off-by: zane-neo <zaniu@amazon.com> --------- Signed-off-by: zane-neo <zaniu@amazon.com>

…x the issue. (opensearch-project#16377) Signed-off-by: zane-neo <zaniu@amazon.com>

…ld (opensearch-project#15258) * fix cluster not able to spin up issue when disk usage exceeds threshold Signed-off-by: zane-neo <zaniu@amazon.com> * Add comment to changes Signed-off-by: zane-neo <zaniu@amazon.com> * Add UT to ensure the keepAliveThread starts before node starts Signed-off-by: zane-neo <zaniu@amazon.com> * remove unused imports Signed-off-by: zane-neo <zaniu@amazon.com> * Fix forbidden API calls check failed issue Signed-off-by: zane-neo <zaniu@amazon.com> * format code Signed-off-by: zane-neo <zaniu@amazon.com> * format code Signed-off-by: zane-neo <zaniu@amazon.com> * change setInstance method to static Signed-off-by: zane-neo <zaniu@amazon.com> * Add countdownlatch in test to coordinate the thread to avoid concureency issue caused test failure Signed-off-by: zane-neo <zaniu@amazon.com> --------- Signed-off-by: zane-neo <zaniu@amazon.com>

…x the issue. (opensearch-project#16377) Signed-off-by: zane-neo <zaniu@amazon.com>

zane-neo mentioned this pull request Aug 15, 2024

[BUG] OpenSearch not starting - [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block]; #14791

Open

zane-neo force-pushed the fix-cluster-unable-spin-up branch from 7ce9886 to 3e8df68 Compare August 16, 2024 06:35

zane-neo marked this pull request as ready for review August 16, 2024 06:36

zane-neo requested review from anasalkouz, andrross, ashking94, Bukhtawar, CEHENKLE, dblock, dbwiddis, gbbafna, kotwanikunal, linuxpi, mch2, msfroh, nknize, owaiskazi19, reta, Rishikesh1159, sachinpkale, saratvemulapalli, shwetathareja, sohami and VachaShah as code owners August 16, 2024 06:36

zane-neo changed the title ~~draft PR to fix cluster not able to spin up issue when disk usage exceeds threshold~~ fix cluster not able to spin up issue when disk usage exceeds threshold Aug 16, 2024

zane-neo added 6 commits October 16, 2024 14:13

Add UT to ensure the keepAliveThread starts before node starts

4fcd310

Signed-off-by: zane-neo <zaniu@amazon.com>

remove unused imports

2ad0126

Signed-off-by: zane-neo <zaniu@amazon.com>

Fix forbidden API calls check failed issue

acebaf3

Signed-off-by: zane-neo <zaniu@amazon.com>

format code

c29a02b

Signed-off-by: zane-neo <zaniu@amazon.com>

format code

2b3cd47

Signed-off-by: zane-neo <zaniu@amazon.com>

change setInstance method to static

af9d70d

Signed-off-by: zane-neo <zaniu@amazon.com>

zane-neo force-pushed the fix-cluster-unable-spin-up branch from c81fbc1 to af9d70d Compare October 16, 2024 06:13

Add countdownlatch in test to coordinate the thread to avoid concuree…

7f24452

…ncy issue caused test failure Signed-off-by: zane-neo <zaniu@amazon.com>

dblock approved these changes Oct 16, 2024

View reviewed changes

dblock merged commit 62081f2 into opensearch-project:main Oct 16, 2024
38 checks passed

opensearch-trigger-bot bot mentioned this pull request Oct 16, 2024

[Backport 2.x] fix cluster not able to spin up issue when disk usage exceeds threshold #16351

Closed

zane-neo mentioned this pull request Oct 18, 2024

[BUG] Observability plugin's system index creating is impacting on OpenSearch node starts opensearch-project/observability#1872

Open

zane-neo added a commit to zane-neo/OpenSearch that referenced this pull request Oct 18, 2024

Revert "fix cluster not able to spin up issue when disk usage exceeds…

fef7477

… threshold (opensearch-project#15258)" This reverts commit 62081f2.

zane-neo added a commit to zane-neo/OpenSearch that referenced this pull request Oct 18, 2024

Revert "fix cluster not able to spin up issue when disk usage exceeds…

916c50c

… threshold (opensearch-project#15258)" This reverts commit 62081f2.

zane-neo mentioned this pull request Oct 18, 2024

Revert #15258 to figure out a better approach to fix the issue. #16377

Merged

3 tasks

zane-neo added a commit to zane-neo/OpenSearch that referenced this pull request Oct 18, 2024

Revert opensearch-project#15258 to figure out a better approach to fi…

9ad7109

…x the issue. Signed-off-by: zane-neo <zaniu@amazon.com>

dbwiddis pushed a commit that referenced this pull request Oct 18, 2024

Revert #15258 to figure out a better approach to fix the issue. (#16377)

0bded88

Signed-off-by: zane-neo <zaniu@amazon.com>

dk2k pushed a commit to dk2k/OpenSearch that referenced this pull request Oct 21, 2024

Revert opensearch-project#15258 to figure out a better approach to fi…

f4f37d5

…x the issue. (opensearch-project#16377) Signed-off-by: zane-neo <zaniu@amazon.com>

opensearch-ci-bot mentioned this pull request Oct 23, 2024

[AUTOCUT] Gradle Check Flaky Test Report for UpdateByQueryBasicTests #16439

Open

akolarkunnu pushed a commit to akolarkunnu/OpenSearch that referenced this pull request Jan 21, 2025

Revert opensearch-project#15258 to figure out a better approach to fi…

cc95e72

…x the issue. (opensearch-project#16377) Signed-off-by: zane-neo <zaniu@amazon.com>

opensearch-ci-bot mentioned this pull request Jan 29, 2025

[AUTOCUT] Gradle Check Flaky Test Report for RemoteStoreRefreshListenerTests #17174

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix cluster not able to spin up issue when disk usage exceeds threshold #15258

fix cluster not able to spin up issue when disk usage exceeds threshold #15258

zane-neo commented Aug 15, 2024 •

edited

Loading

github-actions bot commented Aug 15, 2024

dblock commented Aug 15, 2024 •

edited

Loading

zane-neo commented Aug 16, 2024

github-actions bot commented Aug 16, 2024

github-actions bot commented Aug 16, 2024

github-actions bot commented Oct 16, 2024

github-actions bot commented Oct 16, 2024

github-actions bot commented Oct 16, 2024

dblock left a comment

dblock commented Oct 18, 2024

fix cluster not able to spin up issue when disk usage exceeds threshold #15258

fix cluster not able to spin up issue when disk usage exceeds threshold #15258

Conversation

zane-neo commented Aug 15, 2024 • edited Loading

Description

Related Issues

Check List

github-actions bot commented Aug 15, 2024

dblock commented Aug 15, 2024 • edited Loading

zane-neo commented Aug 16, 2024

github-actions bot commented Aug 16, 2024

github-actions bot commented Aug 16, 2024

github-actions bot commented Oct 16, 2024

github-actions bot commented Oct 16, 2024

github-actions bot commented Oct 16, 2024

dblock left a comment

Choose a reason for hiding this comment

dblock commented Oct 18, 2024

zane-neo commented Aug 15, 2024 •

edited

Loading

dblock commented Aug 15, 2024 •

edited

Loading