[Backport] [Segment Replication] Added mixed and rolling upgrade bwc test (#7537)

[dreamer-89]

Manual backport to 2.x for below PRs

1. [Segment Replication] Added mixed and rolling upgrade bwc test #7537
2. [Segment Replication] Wait for shard relocation before building node to shard allocation map #7876
3. [Segment Replication] Update segrep bwc tests to verify replica checkpoints and skip tests for 1.x bwc versions #8203
4. [Segment Replication] Update bwc test to rely on segments for verification #8267

Resolves #8202

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/15699/
• CommitID: fe81780
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[dreamer-89]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/15699/
• CommitID: fe81780
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

The test currently fails as segment replication is not available for 1.x branch. One possible hack would be to check node version for all nodes and return in case any node running on 1.x but it is not clean. @reta : Would you know a way around this ?

REPRODUCE WITH: ./gradlew ':qa:rolling-upgrade:v1.3.10#oldClusterTest' --tests "org.opensearch.upgrades.IndexingIT.testIndexingWithSegRep" -Dtests.seed=C8DFD73DC280DA80 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=it-IT -Dtests.timezone=Pacific/Truk -Druntime.java=17

org.opensearch.upgrades.IndexingIT > testIndexingWithSegRep FAILED
    org.opensearch.client.ResponseException: method [PUT], host [http://127.0.0.1:35219/], URI [/test-index-segrep], status line [HTTP/1.1 400 Bad Request]
    {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"unknown setting [index.replication.type] please check that any required plugins are installed, or check the breaking changes documentation for removed settings"}],"type":"illegal_argument_exception","reason":"unknown setting [index.replication.type] please check that any required plugins are installed, or check the breaking changes documentation for removed settings"},"status":400}
        at __randomizedtesting.SeedInfo.seed([C8DFD73DC280DA80:47A6C17036D92F41]:0)
        at app//org.opensearch.client.RestClient.convertResponse(RestClient.java:375)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:345)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:320)
        at app//org.opensearch.test.rest.OpenSearchRestTestCase.createIndex(OpenSearchRestTestCase.java:981)
        at app//org.opensearch.test.rest.OpenSearchRestTestCase.createIndex(OpenSearchRestTestCase.java:964)
        at app//org.opensearch.test.rest.OpenSearchRestTestCase.createIndex(OpenSearchRestTestCase.java:960)
        at app//org.opensearch.upgrades.IndexingIT.testIndexingWithSegRep(IndexingIT.java:257)

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/18037/
• CommitID: 58830ea
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[dreamer-89]

Hit couple of issues while working on backport PR.

1. Compilation errors on JDK 11 due to bug on JDK. The fix was available on 11.0.17+ version

 [2.9.0] > Task :server:compileJava
 [2.9.0] Note: Recompile with -Xlint:deprecation for details.
 [2.9.0] Note: Some input files use unchecked or unsafe operations.
 [2.9.0] Note: Recompile with -Xlint:unchecked for details.
 [2.9.0] compiler message file broken: key=compiler.misc.msg.bug arguments=11, {1}, {2}, {3}, {4}, {5}, {6}, {7}
 [2.9.0] java.lang.NullPointerException
 [2.9.0]        at jdk.compiler/com.sun.tools.javac.comp.Flow$FlowAnalyzer.visitApply(Flow.java:1235)
 [2.9.0]        at jdk.compiler/com.sun.tools.javac.tree.JCTree$JCMethodInvocation.accept(JCTree.java:1634)
 [2.9.0]        at jdk.compiler/com.sun.tools.javac.tree.TreeScanner.scan(TreeScanner.java:49)
 [2.9.0]        at jdk.compiler/com.sun.tools.javac.comp.Flow$BaseAnalyzer.scan(Flow.java:398)
 [2.9.0]        at jdk.compiler/com.sun.tools.javac.tree.TreeScanner.visitExec(TreeScanner.java:213)

2. Response object parsing with ObjectPath resulting in NPE. The issue happens when index _stats API response is parsed here. I hit same issue wherever Response parsing was attempted e.g. FullClusterRestartIT. I see the issue never happens on CI so probably some issue with configuration. This is not yet root caused.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/18253/
• CommitID: 7a08dea
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/18417/
• CommitID: 93ed0cf
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[dreamer-89]

PR is blocked due to #8322 where replica shard segment replication stats (_cat/segment_replication) are not updated on primary.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/18597/
• CommitID: 51397d5
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[dreamer-89]

This PR was failing because of breaking change introduced in #8020. Opened issue #8322 for tracking.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/19970/
• CommitID: 8fc0515
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/19977/
• CommitID: 609e662
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/20072/
• CommitID: 8005cb6
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.index.SegmentReplicationPressureIT.testBelowReplicaLimit
      1 org.opensearch.action.admin.cluster.node.tasks.ResourceAwareTasksTests.testTaskResourceTrackingDuringTaskCancellation

• URL: https://build.ci.opensearch.org/job/gradle-check/20074/
• CommitID: 3df7b6b
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov]

Codecov Report

Merging #7626 (3df7b6b) into 2.x (4a62494) will decrease coverage by 0.08%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##                2.x    #7626      +/-   ##
============================================
- Coverage     70.69%   70.62%   -0.08%     
+ Complexity    57214    57205       -9     
============================================
  Files          4747     4747              
  Lines        270933   270933              
  Branches      39975    39975              
============================================
- Hits         191544   191340     -204     
- Misses        62909    63208     +299     
+ Partials      16480    16385      -95     

see 441 files with indirect coverage changes