Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Snapshot V2] Support pinned timestamp in clone flow #15524

Merged
merged 26 commits into from
Sep 6, 2024

Conversation

anshu1106
Copy link
Contributor

@anshu1106 anshu1106 commented Aug 30, 2024

Description

This PR introduce changes to support pinned timestamp in clone flow.

  • The clone operations pins the same timestamp that is pinned by the source snapshot
  • clone is supported only for * indices similar to create snapshot v2 where we create snapshot using v2 only if indices list is empty in the request ie all the indices are snapshotted
  • Unlike traditional clone operations, the v2 clone performs a synchronous clone execution and doesn't
    upload any shard metadata to the snapshot repository.
  • The pinned timestamp is later reconciled with remote store segment and translog metadata files during the restore
    operation.

Related Issues

Resolves #15316

Check List

  • Functionality includes testing.
  • API changes companion pull request created, if applicable.
  • Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>
@github-actions github-actions bot added enhancement Enhancement or improvement to existing feature or request Storage:Remote Storage:Snapshots labels Aug 30, 2024
Copy link
Contributor

❌ Gradle check result for 5b21150: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>
Copy link
Contributor

❌ Gradle check result for 83f699c: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Copy link

codecov bot commented Sep 5, 2024

Codecov Report

Attention: Patch coverage is 0% with 122 lines in your changes missing coverage. Please review.

Project coverage is 71.73%. Comparing base (729e40d) to head (4f694d5).
Report is 22 commits behind head on main.

Files with missing lines Patch % Lines
...ava/org/opensearch/snapshots/SnapshotsService.java 0.00% 121 Missing ⚠️
.../snapshots/clone/TransportCloneSnapshotAction.java 0.00% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main   #15524      +/-   ##
============================================
- Coverage     71.95%   71.73%   -0.23%     
+ Complexity    64192    64094      -98     
============================================
  Files          5270     5271       +1     
  Lines        300052   300287     +235     
  Branches      43368    43387      +19     
============================================
- Hits         215917   215421     -496     
- Misses        66442    67176     +734     
+ Partials      17693    17690       -3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Collaborator

@Bukhtawar Bukhtawar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like we need more tests?

Sorry , i missed this before merging . @anshu1106 , can you please take this as a follow up ?

@gbbafna gbbafna merged commit debd040 into opensearch-project:main Sep 6, 2024
32 of 34 checks passed
opensearch-trigger-bot bot pushed a commit that referenced this pull request Sep 6, 2024
Signed-off-by: Anshu Agarwal <anshukag@amazon.com>
(cherry picked from commit debd040)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@anshu1106
Copy link
Contributor Author

Looks like we need more tests?

Sorry , i missed this before merging . @anshu1106 , can you please take this as a follow up ?

ITs are covering all the cases. Let me know if you have anything specific.

ashking94 pushed a commit that referenced this pull request Sep 6, 2024
(cherry picked from commit debd040)

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@gbbafna gbbafna added the backport 2.x Backport to 2.x branch label Sep 6, 2024
opensearch-trigger-bot bot pushed a commit that referenced this pull request Sep 6, 2024
Signed-off-by: Anshu Agarwal <anshukag@amazon.com>
(cherry picked from commit debd040)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
sachinpkale pushed a commit that referenced this pull request Sep 9, 2024
Signed-off-by: Anshu Agarwal <anshukag@amazon.com>
(cherry picked from commit debd040)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
sachinpkale pushed a commit that referenced this pull request Sep 9, 2024
(cherry picked from commit debd040)

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
akolarkunnu pushed a commit to akolarkunnu/OpenSearch that referenced this pull request Sep 10, 2024
dk2k pushed a commit to dk2k/OpenSearch that referenced this pull request Oct 16, 2024
dk2k pushed a commit to dk2k/OpenSearch that referenced this pull request Oct 17, 2024
dk2k pushed a commit to dk2k/OpenSearch that referenced this pull request Oct 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport 2.x Backport to 2.x branch backport 2.17 enhancement Enhancement or improvement to existing feature or request skip-changelog Storage:Remote Storage:Snapshots
Projects
Status: ✅ Done
Development

Successfully merging this pull request may close these issues.

[Snapshot V2] Integrate clone snapshot flow with pinned timestamp
4 participants