[Remote Store][Refactor] Introduce RemoteStoreRestoreService to handle restore from remote store #9256

linuxpi · 2023-08-11T06:53:47Z

Description

Currently remote store backed index restore logic resides in RestoreService along with snapshot restore logic.
We introduce a new RemoteStoreRestoreService which will host the logic for remote store restore.
RestoreService will keep on existing and will only contain logic around snapshot restore.

Related Issues

NA

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

server/src/main/java/org/opensearch/index/recovery/RemoteStoreRestoreService.java

github-actions · 2023-08-11T07:02:32Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/22441/
CommitID: 91b3af6
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

opensearch-trigger-bot · 2023-08-11T07:24:25Z

Compatibility status:



> Task :checkCompatibility
Incompatible components: [https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git]
Compatible components: [https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/performance-analyzer-rca.git]

BUILD SUCCESSFUL in 30m 11s

opensearch-trigger-bot · 2023-08-11T07:27:16Z

Compatibility status:



> Task :checkCompatibility
Incompatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/performance-analyzer.git]
Compatible components: [https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

BUILD SUCCESSFUL in 28m 29s

github-actions · 2023-08-11T07:41:48Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/22442/
CommitID: 425722c

codecov · 2023-08-11T07:42:46Z

Codecov Report

Merging #9256 (425722c) into main (1162865) will decrease coverage by 0.03%.
Report is 3 commits behind head on main.
The diff coverage is 12.85%.

❗ Current head 425722c differs from pull request most recent head 0d07bb0. Consider uploading reports for the commit 0d07bb0 to get more accurate results

@@             Coverage Diff              @@
##               main    #9256      +/-   ##
============================================
- Coverage     71.03%   71.01%   -0.03%     
+ Complexity    57318    57313       -5     
============================================
  Files          4770     4767       -3     
  Lines        270540   270575      +35     
  Branches      39561    39563       +2     
============================================
- Hits         192186   192146      -40     
- Misses        62147    62233      +86     
+ Partials      16207    16196      -11

Files Changed	Coverage Δ
...ore/restore/TransportRestoreRemoteStoreAction.java	`26.66% <0.00%> (+1.66%)`	⬆️
...arch/index/recovery/RemoteStoreRestoreService.java	`8.19% <8.19%> (ø)`
server/src/main/java/org/opensearch/node/Node.java	`86.10% <100.00%> (+0.05%)`	⬆️
.../java/org/opensearch/snapshots/RestoreService.java	`57.18% <100.00%> (ø)`

... and 528 files with indirect coverage changes

opensearch-trigger-bot · 2023-08-11T09:53:58Z

Compatibility status:



> Task :checkCompatibility
Incompatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/performance-analyzer.git]
Compatible components: [https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

BUILD SUCCESSFUL in 24m 33s

github-actions · 2023-08-11T10:00:21Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/22448/
CommitID: c57fd7f
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

…e store Signed-off-by: bansvaru <bansvaru@amazon.com>

Signed-off-by: bansvaru <bansvaru@amazon.com>

github-actions · 2023-08-11T11:25:40Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/22450/
CommitID: 4d85d01
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

opensearch-trigger-bot · 2023-08-11T11:34:32Z

Compatibility status:



> Task :checkCompatibility
Incompatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git]
Compatible components: [https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

BUILD SUCCESSFUL in 22m 6s

opensearch-trigger-bot · 2023-08-11T11:40:06Z

Compatibility status:



> Task :checkCompatibility
Incompatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/performance-analyzer.git]
Compatible components: [https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

BUILD SUCCESSFUL in 21m 47s

opensearch-trigger-bot · 2023-08-11T11:42:27Z

Compatibility status:



> Task :checkCompatibility
Incompatible components: [https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git]
Compatible components: [https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/performance-analyzer-rca.git]

BUILD SUCCESSFUL in 21m 59s

github-actions · 2023-08-11T11:51:16Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/22449/
CommitID: 98a5ccb
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-08-11T12:01:55Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/22451/
CommitID: 0d07bb0
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

opensearch-trigger-bot · 2023-08-11T12:55:46Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-9256-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 d102a04b6eecf994527c7bd1bf75a1584bfabb1a
# Push it to GitHub
git push --set-upstream origin backport/backport-9256-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-9256-to-2.x.

…from remote store (opensearch-project#9256) Signed-off-by: bansvaru <bansvaru@amazon.com>

…from remote store (opensearch-project#9256) Signed-off-by: bansvaru <bansvaru@amazon.com> (cherry picked from commit d102a04)

…ice. Ref opensearch-project#9256 Signed-off-by: bansvaru <bansvaru@amazon.com>

…from remote store (opensearch-project#9256) Signed-off-by: bansvaru <bansvaru@amazon.com>

…from remote store (opensearch-project#9256) Signed-off-by: bansvaru <bansvaru@amazon.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com>

…from remote store (opensearch-project#9256) Signed-off-by: bansvaru <bansvaru@amazon.com> Signed-off-by: Ivan Brusic <ivan.brusic@flocksafety.com>

…from remote store (opensearch-project#9256) Signed-off-by: bansvaru <bansvaru@amazon.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com>

linuxpi commented Aug 11, 2023

View reviewed changes

server/src/main/java/org/opensearch/index/recovery/RemoteStoreRestoreService.java Show resolved Hide resolved

linuxpi changed the title ~~Introduce RemoteStoreRestoreService to handle all restores from remot…~~ [Remote Store][Refactor] Introduce RemoteStoreRestoreService to handle all restores from remote store Aug 11, 2023

linuxpi changed the title ~~[Remote Store][Refactor] Introduce RemoteStoreRestoreService to handle all restores from remote store~~ [Remote Store][Refactor] Introduce RemoteStoreRestoreService to handle restore from remote store Aug 11, 2023

linuxpi marked this pull request as ready for review August 11, 2023 08:53

linuxpi requested review from reta, anasalkouz, andrross, Bukhtawar, CEHENKLE, dblock, gbbafna, setiah, kartg, kotwanikunal, mch2, nknize, owaiskazi19, Rishikesh1159, ryanbogan, saratvemulapalli, shwetathareja, dreamer-89, tlfeng, VachaShah and dbwiddis as code owners August 11, 2023 08:53

linuxpi added 4 commits August 11, 2023 16:47

Introduce RemoteStoreRestoreService to handle all restores from remot…

1d7dcf7

…e store Signed-off-by: bansvaru <bansvaru@amazon.com>

Fix spotless errors

5b20116

Signed-off-by: bansvaru <bansvaru@amazon.com>

fix docstring

c955d7d

Signed-off-by: bansvaru <bansvaru@amazon.com>

Empty Commit

4d85d01

Signed-off-by: bansvaru <bansvaru@amazon.com>

linuxpi force-pushed the remote-store-restore-service branch from 98a5ccb to 4d85d01 Compare August 11, 2023 11:17

fix import path for ActionListener

0d07bb0

Signed-off-by: bansvaru <bansvaru@amazon.com>

gbbafna approved these changes Aug 11, 2023

View reviewed changes

gbbafna merged commit d102a04 into opensearch-project:main Aug 11, 2023

gbbafna added the backport 2.x Backport to 2.x branch label Aug 11, 2023

gbbafna deleted the remote-store-restore-service branch August 11, 2023 12:54

neetikasinghal pushed a commit to neetikasinghal/OpenSearch that referenced this pull request Aug 11, 2023

[Remote Store] Introduce RemoteStoreRestoreService to handle restore …

11e5689

…from remote store (opensearch-project#9256) Signed-off-by: bansvaru <bansvaru@amazon.com>

linuxpi mentioned this pull request Aug 11, 2023

[Backport 2.x][Remote Store] Introduce RemoteStoreRestoreService to handle restore … #9261

Merged

6 tasks

linuxpi added a commit to linuxpi/OpenSearch that referenced this pull request Aug 14, 2023

[Remote Store] Remove stale code for remote restore from Restore Serv…

1e87cd7

…ice. Ref opensearch-project#9256 Signed-off-by: bansvaru <bansvaru@amazon.com>

linuxpi mentioned this pull request Aug 14, 2023

[Remote Store] Remove stale code for remote restore from Restore Serv… #9293

Merged

6 tasks

linuxpi added a commit to linuxpi/OpenSearch that referenced this pull request Aug 14, 2023

[Remote Store] Introduce RemoteStoreRestoreService to handle restore …

30cf1e8

…from remote store (opensearch-project#9256) Signed-off-by: bansvaru <bansvaru@amazon.com>

linuxpi added a commit to linuxpi/OpenSearch that referenced this pull request Aug 16, 2023

[Remote Store] Introduce RemoteStoreRestoreService to handle restore …

39e31e9

…from remote store (opensearch-project#9256) Signed-off-by: bansvaru <bansvaru@amazon.com>

gbbafna mentioned this pull request Aug 8, 2024

Add Varun Bansal as maintainer #15163

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Remote Store][Refactor] Introduce RemoteStoreRestoreService to handle restore from remote store #9256

[Remote Store][Refactor] Introduce RemoteStoreRestoreService to handle restore from remote store #9256

linuxpi commented Aug 11, 2023 •

edited

Loading

github-actions bot commented Aug 11, 2023

opensearch-trigger-bot bot commented Aug 11, 2023

opensearch-trigger-bot bot commented Aug 11, 2023

github-actions bot commented Aug 11, 2023

codecov bot commented Aug 11, 2023 •

edited

Loading

opensearch-trigger-bot bot commented Aug 11, 2023

github-actions bot commented Aug 11, 2023

github-actions bot commented Aug 11, 2023

opensearch-trigger-bot bot commented Aug 11, 2023

opensearch-trigger-bot bot commented Aug 11, 2023

opensearch-trigger-bot bot commented Aug 11, 2023

github-actions bot commented Aug 11, 2023

github-actions bot commented Aug 11, 2023

opensearch-trigger-bot bot commented Aug 11, 2023

[Remote Store][Refactor] Introduce RemoteStoreRestoreService to handle restore from remote store #9256

[Remote Store][Refactor] Introduce RemoteStoreRestoreService to handle restore from remote store #9256

Conversation

linuxpi commented Aug 11, 2023 • edited Loading

Description

Related Issues

Check List

github-actions bot commented Aug 11, 2023

Gradle Check (Jenkins) Run Completed with:

opensearch-trigger-bot bot commented Aug 11, 2023

opensearch-trigger-bot bot commented Aug 11, 2023

github-actions bot commented Aug 11, 2023

Gradle Check (Jenkins) Run Completed with:

codecov bot commented Aug 11, 2023 • edited Loading

Codecov Report

opensearch-trigger-bot bot commented Aug 11, 2023

github-actions bot commented Aug 11, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 11, 2023

Gradle Check (Jenkins) Run Completed with:

opensearch-trigger-bot bot commented Aug 11, 2023

opensearch-trigger-bot bot commented Aug 11, 2023

opensearch-trigger-bot bot commented Aug 11, 2023

github-actions bot commented Aug 11, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 11, 2023

Gradle Check (Jenkins) Run Completed with:

opensearch-trigger-bot bot commented Aug 11, 2023

linuxpi commented Aug 11, 2023 •

edited

Loading

codecov bot commented Aug 11, 2023 •

edited

Loading