Exclude OpenSearch system index when IT cleanup

[dai-chen]

Description

Many IT failed in 2.6 release test due to no permission to delete system index .opensearch-observability. Fixed IT base class to exclude .opensearch-xxx index when IT cleanup. Meanwhile as quick fix, ignore exception just in case system index with different naming.

Suite: Test class org.opensearch.sql.legacy.AggregationExpressionIT
  2> org.opensearch.client.ResponseException: method [DELETE], host [https://localhost:9200], URI [/.opensearch-observability], status line [HTTP/1.1 403 Forbidden]
    {"error":{"root_cause":[{"type":"security_exception","reason":"no permissions for [] and User [name=admin, backend_roles=[admin], requestedTenant=null]"}],"type":"security_exception","reason":"no permissions for [] and User [name=admin, backend_roles=[admin], requestedTenant=null]"},"status":403}
        at __randomizedtesting.SeedInfo.seed([917732C6B2C1F0C0]:0)
        at app//org.opensearch.client.RestClient.convertResponse(RestClient.java:375)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:345)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:320)
        at app//org.opensearch.sql.legacy.OpenSearchSQLRestTestCase.wipeAllOpenSearchIndices(OpenSearchSQLRestTestCase.java:78)
        at app//org.opensearch.sql.legacy.SQLIntegTestCase.cleanUpIndices(SQLIntegTestCase.java:148)
...

Publishing this to 2.6 branch directly to avoid blocked by compile error in main branch.

TODO

System index may not have same prefix name: https://github.com/opensearch-project/security/blob/2df8acd6789910ced83d6a198b3a06fa221fb58e/tools/install_demo_configuration.sh#L380. The reason we have to include hidden/system index here is probably because some IT are testing with hidden index: opendistro-for-elasticsearch/sql#758. For permanent fix, one option maybe using same and unique prefix name for test index used in all SQL IT. So we only need to clean up these related to us.

Issues Resolved

N/A

Check List

• New functionality includes testing.
  • All tests pass, including unit test, integration test and doctest
• New functionality has been documented.
  • New functionality has javadoc added
  • New functionality has user manual doc added
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[opensearch-trigger-bot]

The backport to main failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-main main
# Navigate to the new working tree
cd .worktrees/backport-main
# Create a new branch
git switch --create backport/backport-1380-to-main
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 71d860410efa00a44290be030a9e148b29cd4fe9
# Push it to GitHub
git push --set-upstream origin backport/backport-1380-to-main
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-main

Then, create a pull request where the base branch is main and the compare/head branch is backport/backport-1380-to-main.