[BUG] CreateRemoteIndexIT/CreateRemoteIndexTranslogDisabledIT failures

[andrross]

Multiple failures here (https://build.ci.opensearch.org/job/gradle-check/13937/) that succeeded on retry

      1 org.opensearch.remotestore.CreateRemoteIndexTranslogDisabledIT.classMethod
      1 org.opensearch.remotestore.CreateRemoteIndexIT.testRemoteStoreDisabledByUser
      1 org.opensearch.remotestore.CreateRemoteIndexIT.classMethod

/cc @linuxpi Looks like these are tests that you recently added. Can you take a look?

[andrross]

More failures here: #7125 (comment)

[linuxpi]

Multiple failures here (https://build.ci.opensearch.org/job/gradle-check/13937/) that succeeded on retry

      1 org.opensearch.remotestore.CreateRemoteIndexTranslogDisabledIT.classMethod
      1 org.opensearch.remotestore.CreateRemoteIndexIT.testRemoteStoreDisabledByUser
      1 org.opensearch.remotestore.CreateRemoteIndexIT.classMethod

/cc @linuxpi Looks like these are tests that you recently added. Can you take a look?

Sure, let me check and fix

[linuxpi]

The failure are because we are not cleaning up snapshot dir.

java.lang.AssertionError: [test-idx-1][2], node[vX6f0DTmRfC1il8YDrSIkQ], [P], s[STARTED], a[id=53rPm8H3SrepttaigUli0A] has unreleased snapshotted index commits

java.lang.RuntimeException: file handle leaks: [InputStream(/var/jenkins/workspace/gradle-check/search/server/build/testrun/internalClusterTest/temp/org.opensearch.remotestore.CreateRemoteIndexIT_58A3D614EB68203F-001/tempDir-008/repos/fgUzGkzgHm/qWmYzVJSTKeMhahfEsSuhw/0/segments/data/segment_infos_snapshot_filename__3__PXCpdocBnatMeQEWYB27)]

Constantly failing with following:

./gradlew ':server:internalClusterTest' --tests "org.opensearch.remotestore.CreateRemoteIndexIT" -Dtests.seed=58A3D614EB68203F -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=en -Dtests.timezone=Etc/UTC -Dtests.iters=20

./gradlew ':server:internalClusterTest' --tests "org.opensearch.remotestore.CreateRemoteIndexTranslogDisabledIT" -Dtests.seed=58A3D614EB68203F -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=en -Dtests.timezone=Etc/UTC -Dtests.iters=20


./gradlew ':server:internalClusterTest' --tests "org.opensearch.remotestore.CreateRemoteIndexClusterDefaultDocRep" -Dtests.seed=58A3D614EB68203F -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=en -Dtests.timezone=Etc/UTC -Dtests.iters=20

#7141 fixes this flakyness