Added insecureString options to allow to easier migration to secure setting variants

[chriswhite199]

Description

To address discussion points in the following issues and PRs in common-utils and security plugin:

• [FEATURE] Update password settings to utilize secure variants common-utils#334
• Added secure settings for ssl related passwords: security#2296

This change adds override options to SecureSetting.insecureString such that migrating from legacy insecure settings to secure variants can be achieved without breaking existing cluster deployments and having to set opensearch.allow_insecure_settings. Instead usage of deprecated insecure settings are logged (warn level) with a note as to the new secure variant of the setting too.

If approved, changes can be made to common-utils and security plugin to utilize this new code

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/7819/
• CommitID: f591a72

[codecov-commenter]

Codecov Report

Merging #5496 (d39d86b) into main (606255f) will decrease coverage by 0.03%.
The diff coverage is 84.61%.

❗ Current head d39d86b differs from pull request most recent head c547046. Consider uploading reports for the commit c547046 to get more accurate results

@@             Coverage Diff              @@
##               main    #5496      +/-   ##
============================================
- Coverage     70.85%   70.82%   -0.03%     
- Complexity    56553    58693    +2140     
============================================
  Files          4722     4769      +47     
  Lines        267594   280696   +13102     
  Branches      39212    40535    +1323     
============================================
+ Hits         189600   198801    +9201     
- Misses        61983    65563    +3580     
- Partials      16011    16332     +321     

Impacted Files	Coverage Δ
.../org/opensearch/common/settings/SecureSetting.java	74.62% <84.61%> (+9.71%)	⬆️

... and 1548 files with indirect coverage changes

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/7839/
• CommitID: dcfc029
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/7841/
• CommitID: d37d713
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/7847/
• CommitID: 1feb2cd
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/7853/
• CommitID: 2fe4347

[dblock]

Should we be deprecating insecure settings instead rather than just warn-usage?

[saratvemulapalli]

We could put out the warning in 2.x line and probably deprecate it in 3.x?

[chriswhite199]

Should we be deprecating insecure settings instead rather than just warn-usage?

InsecureStringSettings are already marked as deprecated and a log entry is added when used in settings, but that doesn't give developers to opportunity to flag in the log entry what they should be using instead.

The issue discussed over in security plugin was related to not breaking existing setups and plugins that might still expect the insecure setting to be used. Just using insecureString wouldn't suffice as unless they had allow_insecure_settings to be true, startup would fail.

[dblock]

Last question: are we going to flood logs with these? should the warning happen once per node/constructor?

[chriswhite199]

Last question: are we going to flood logs with these? should the warning happen once per node/constructor?

I don't think the logs will be flooded, but any time the settings is queried it will log yes. Typically this only occurs at startup for a plugin, but I see the risk where a plugin might repeatedly read the setting (as say part of a custom mapping).

The issue with construction time is that we don't yet know whether the insecure setting is actually used in configuration. It should be trivial to add a volatile or sync block around the current log statement and only log it once.

[dblock]

Add a test that it's not logged twice in a separate test.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/8085/
• CommitID: dffecd6
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/8086/
• CommitID: 0f6f812
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[dreamer-89]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/8086/
• CommitID: 0f6f812
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

Failing due to version bump after 2.4.1 release, #5560. Fixed is merged into main branch. @chriswhite199: Can you please rebase your branch against latest main branch ?

* What went wrong:
Execution failed for task ':distribution:bwc:bugfix:buildBwcLinuxTar'.
> Building 2.4.1 didn't generate expected file /var/jenkins/workspace/gradle-check/search/distribution/bwc/bugfix/build/bwc/checkout-2.4/distribution/archives/linux-tar/build/distributions/opensearch-min-2.4.1-SNAPSHOT-linux-x64.tar.gz

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/8116/
• CommitID: 0f6f812
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/8168/
• CommitID: 87280ef

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/9483/
• CommitID: d39d86b

[kartg]

Looks like this has been ready to merge for a while but got missed out amongst our other PRs. Since the baseline is quite dated, i'll rebase from the current main and resolve the CHANGELOG conflict

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/17625/
• CommitID: 54ce8de
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/17711/
• CommitID: 54ce8de
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      2 org.opensearch.remotestore.RemoteStoreRefreshListenerIT.testRemoteRefreshRetryOnFailure
      1 org.opensearch.search.backpressure.SearchBackpressureIT.testSearchShardTaskCancellationWithHighCpu
      1 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testPitCreatedOnReplica

• URL: https://build.ci.opensearch.org/job/gradle-check/17729/
• CommitID: c547046
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-5496-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 0b775e79f637c7225ae908e368f1a62ce9ece011
# Push it to GitHub
git push --set-upstream origin backport/backport-5496-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-5496-to-2.x.