[Remote Segment Store] Make metadata file immutable

[sachinpkale]

Description

• Currently, metadata file in remote segment store is overwritten at each refresh and new file is created only at commit.
• This was done to keep the number of metadata files limited so that reading latest metadata file will be quick.
• But this can create concurrency issues as the file that is getting uploaded and the file that is getting read can be different due to same name.
• To avoid this, in this PR, we make the remote segment store metadata file immutable. We make sure to upload metadata file with different name at each refresh.
• Following is the format that we are using:

metadata_<Inverted Primary Term>_<Inverted Commit Generation>_<Inverted Translog Generation>

• Here, each inverted number is represented as Long.MAX_VALUE - original number padded with 0s at the front to make the length consistent with that of Long.MAX_VALUE.
• The reason behind using inverted values is to make use of Lexicographic sort order that is supported in majority of the blob stores that remote store feature supports.
• Using translog generation is just to have a strictly increasing counter across nodes in the cluster for a given shard.

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/18744/
• CommitID: e2bcafe
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/18779/
• CommitID: 82fcefd
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/18788/
• CommitID: 6dd3fe0
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/18824/
• CommitID: 9e8bcf8
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/18868/
• CommitID: 018f7d0
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/18869/
• CommitID: 018f7d0
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/18870/
• CommitID: 018f7d0
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/18887/
• CommitID: ff09074
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/18890/
• CommitID: a832868
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/18893/
• CommitID: 6db8294
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/18896/
• CommitID: 7b27a78
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[sachinpkale]

Tagging @harishbhakuni21 to review changes to snapshot interoperability logic as part of this PR.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/18897/
• CommitID: 448d412
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testNodeDropWithOngoingReplication
      1 org.opensearch.index.IndexServiceTests.testAsyncTranslogTrimTaskOnClosedIndex
      1 org.opensearch.cluster.allocation.AwarenessAllocationIT.testThreeZoneOneReplicaWithForceZoneValueAndLoadAwareness

• URL: https://build.ci.opensearch.org/job/gradle-check/18900/
• CommitID: 448d412
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov]

Codecov Report

Merging #8363 (597a712) into main (eca5a6c) will increase coverage by 0.06%.
The diff coverage is 90.74%.

@@             Coverage Diff              @@
##               main    #8363      +/-   ##
============================================
+ Coverage     70.87%   70.93%   +0.06%     
- Complexity    56923    56944      +21     
============================================
  Files          4758     4758              
  Lines        269219   269221       +2     
  Branches      39407    39402       -5     
============================================
+ Hits         190800   190963     +163     
+ Misses        62352    62107     -245     
- Partials      16067    16151      +84     

Impacted Files	Coverage Δ
...opensearch/index/translog/NoOpTranslogManager.java	63.63% <0.00%> (-1.99%)	⬇️
...search/index/shard/RemoteStoreRefreshListener.java	84.84% <66.66%> (-0.21%)	⬇️
...va/org/opensearch/index/store/RemoteDirectory.java	88.37% <87.50%> (-0.52%)	⬇️
...earch/index/store/RemoteSegmentStoreDirectory.java	92.09% <100.00%> (+0.17%)	⬆️
...search/index/translog/InternalTranslogManager.java	67.13% <100.00%> (-4.70%)	⬇️

... and 496 files with indirect coverage changes

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.remotestore.RemoteStoreIT.testStaleCommitDeletionWithoutInvokeFlush

• URL: https://build.ci.opensearch.org/job/gradle-check/19243/
• CommitID: e0823f1
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      2 org.opensearch.remotestore.RemoteStoreIT.testStaleCommitDeletionWithInvokeFlush
      1 org.opensearch.repositories.azure.AzureBlobContainerRetriesTests.testReadNonexistentBlobThrowsNoSuchFileException
      1 org.opensearch.remotestore.RemoteStoreIT.testStaleCommitDeletionWithoutInvokeFlush

• URL: https://build.ci.opensearch.org/job/gradle-check/19251/
• CommitID: 597a712
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-8363-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 5c0e3c94950796f7c99a20456528181ae9f16bbf
# Push it to GitHub
git push --set-upstream origin backport/backport-8363-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-8363-to-2.x.