[BUG] [Remote Store] TranslogCorruptedException in replica recovery if snapshot is restored to new index

[sachinpkale]

Describe the bug

• While restoring a snapshot to an index with same name but different UUID works well for primary shard but replica recovery fails with TranslogCorruptedException.
• This will keep replica shard unassigned and index turns to yellow.

To Reproduce
Steps to reproduce the behavior:

1. Create index and ingest 50 docs
2. Take a snapshot
3. Delete the index
4. Create index with the same name, it will have different UUID
5. Close index
6. Restore from the snapshot

To Mitigate

• Close the index and restore from the snapshot again should solve the issue.

Exception Stacktrace

Caused by: org.opensearch.transport.RemoteTransportException: [node_t0][127.0.0.1:58094][internal:index/shard/recovery/prepare_translog]
Caused by: org.opensearch.index.translog.TranslogCorruptedException: translog from source [/Users/kalsac/Codebase/OpenSearch/server/build/testrun/internalClusterTest/temp/org.opensearch.snapshots.SharedClusterSnapshotRestoreIT_F0EBE56E429E8448-007/tempDir-002/node_t0/nodes/0/indices/LDJA3bMuQoWEJXEbcbQUJw/1/translog/translog-1.tlog] is corrupted, expected shard UUID [33 31 49 64 61 43 42 76 53 6e 4f 4e 35 4c 39 54 69 6b 57 77 67 41] but got: [64 73 73 37 41 7a 75 30 52 64 61 34 53 4b 31 67 59 44 6e 41 4f 51] this translog file belongs to a different translog
    at org.opensearch.index.translog.TranslogHeader.read(TranslogHeader.java:149) ~[main/:?]
    at org.opensearch.index.translog.TranslogReader.open(TranslogReader.java:118) ~[main/:?]
    at org.opensearch.index.translog.Translog.openReader(Translog.java:301) ~[main/:?]
    at org.opensearch.index.translog.Translog.recoverFromFiles(Translog.java:223) ~[main/:?]
    at org.opensearch.index.translog.LocalTranslog.<init>(LocalTranslog.java:76) ~[main/:?]
    at org.opensearch.index.translog.InternalTranslogFactory.newTranslog(InternalTranslogFactory.java:34) ~[main/:?]
    at org.opensearch.index.translog.InternalTranslogManager.openTranslog(InternalTranslogManager.java:362) ~[main/:?]
    at org.opensearch.index.translog.InternalTranslogManager.<init>(InternalTranslogManager.java:67) ~[main/:?]
    at org.opensearch.index.translog.WriteOnlyTranslogManager.<init>(WriteOnlyTranslogManager.java:43) ~[main/:?]
    at org.opensearch.index.engine.NRTReplicationEngine.<init>(NRTReplicationEngine.java:127) ~[main/:?]
    at org.opensearch.test.engine.MockNRTReplicationEngine.<init>(MockNRTReplicationEngine.java:37) ~[framework-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
    at org.opensearch.test.engine.MockEngineFactory.newReadWriteEngine(MockEngineFactory.java:50) ~[framework-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
    at org.opensearch.index.shard.IndexShard.innerOpenEngineAndTranslog(IndexShard.java:2358) ~[main/:?]
    at org.opensearch.index.shard.IndexShard.innerOpenEngineAndTranslog(IndexShard.java:2319) ~[main/:?]
    at org.opensearch.index.shard.IndexShard.openEngineAndSkipTranslogRecovery(IndexShard.java:2301) ~[main/:?]
    at org.opensearch.indices.recovery.RecoveryTarget.lambda$prepareForTranslogOperations$0(RecoveryTarget.java:209) ~[main/:?]