Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Remove] Legacy Version support from Snapshot/Restore Service #4728

Merged
merged 3 commits into from
Oct 19, 2022
Merged

[Remove] Legacy Version support from Snapshot/Restore Service #4728

merged 3 commits into from
Oct 19, 2022

Conversation

nknize
Copy link
Collaborator

@nknize nknize commented Oct 10, 2022 •
edited
Loading

Removes all stale snapshot / restore code for legacy versions prior to OpenSearch 2.0. This includes
handling null ShardGenerations, partial concurrency, null index generations, etc. which are no longer
supported in snapshot versions after legacy 7.5.

relates #2747
depends on #4729

@nknize nknize added non-issue bugs / unexpected behaviors that end up non issues; audit trail simple changes that aren't issues >upgrade Label used when upgrading library dependencies (e.g., Lucene) v3.0.0 Issues and PRs related to version 3.0.0 labels Oct 10, 2022
@nknize nknize requested review from a team and reta as code owners October 10, 2022 19:16
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@nknize
Copy link
Collaborator Author

nknize commented Oct 10, 2022

Failure is fixed in BWC PR #4729. Updated the description to include dependency.

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@nknize nknize force-pushed the remove/ancientSnapshotRestoreLogic branch from 3fc04a5 to f6b96e2 Compare October 12, 2022 04:42
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@nknize
Copy link
Collaborator Author

nknize commented Oct 12, 2022

Reproducible failure unrelated to this PR, exists on main as well. Looks like we ignored these failures because they were flakey for so long, #4162. We'll need to track down the introduction separately:

/gradlew ':server:internalClusterTest' --tests "org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.testRestoreSnapshotAllocationDoesNotExceedWatermark" -Dtests.seed=926BB9577ACEFB6 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=es-PR -Dtests.timezone=America/Virgin -Druntime.java=17

Mismatching shard routings: [[gcbrbsekqi][4], node[CrwBliCDQMGz9KGNyAR9HA], [P], s[STARTED], a[id=9Up1aDjPSy6oha_NNMmmzQ], [gcbrbsekqi][5], node[CrwBliCDQMGz9KGNyAR9HA], [P], s[STARTED], a[id=8tAYnIv3TXi-1uJcCFszmg]]
Expected: a collection with size <1>
     but: collection size was <2>

@nknize nknize force-pushed the remove/ancientSnapshotRestoreLogic branch from f6b96e2 to 8da1f30 Compare October 12, 2022 16:20
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

nknize added 3 commits October 18, 2022 20:51
@nknize
[Remove] Legacy Version support from Snapshot/Restore
7993af2 
Removes all stale snapshot / restore code for legacy versions prior to
OpenSearch 2.0. This includes handling null ShardGenerations, partial
concurrency, null index generations, etc. which are no longer supported in
snapshot versions after legacy 7.5.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
@nknize
add changelog
504506e 
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
@nknize
add back min_version for now
4282407 
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
@nknize nknize force-pushed the remove/ancientSnapshotRestoreLogic branch from 8da1f30 to 4282407 Compare October 19, 2022 01:51
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@nknize nknize merged commit ec34737 into opensearch-project:main Oct 19, 2022
ashking94 pushed a commit to ashking94/OpenSearch that referenced this pull request Nov 7, 2022
@nknize @ashking94
[Remove] Legacy Version support from Snapshot/Restore Service (opense…
2b91668 
…arch-project#4728)

Removes all stale snapshot / restore code for legacy versions prior to
OpenSearch 2.0. This includes handling null ShardGenerations, partial
concurrency, null index generations, etc. which are no longer supported in
snapshot versions after legacy 7.5.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
kartg added a commit to kartg/OpenSearch that referenced this pull request Jan 10, 2023
@kartg
Reintroducing backwards compatibility logic in certain classes
77efe6b 
This reverts changes made in opensearch-project#4728 and opensearch-project#4702. These were only made in main and not backported to 2.x

Signed-off-by: Kartik Ganesh <gkart@amazon.com>
kartg added a commit to kartg/OpenSearch that referenced this pull request Jan 10, 2023
@kartg
Reintroducing backwards compatibility logic in certain classes
08a1128 
This reverts changes made in opensearch-project#4728 and opensearch-project#4702. These were only made in main and not backported to 2.x

Signed-off-by: Kartik Ganesh <gkart@amazon.com>
kartg added a commit to kartg/OpenSearch that referenced this pull request Jan 10, 2023
@kartg
Reintroducing backwards compatibility logic in certain classes
4c96839 
This reverts changes made in opensearch-project#4728 and opensearch-project#4702. These were only made in main and not backported to 2.x

Signed-off-by: Kartik Ganesh <gkart@amazon.com>
kartg added a commit to kartg/OpenSearch that referenced this pull request Jan 11, 2023
@kartg
Reintroducing backwards compatibility logic in certain classes
f63b8cc 
This reverts changes made in opensearch-project#4728 and opensearch-project#4702. These were only made in main and not backported to 2.x
This change also adds unit tests for IndexMetadataGenerations

Signed-off-by: Kartik Ganesh <gkart@amazon.com>
kartg added a commit to kartg/OpenSearch that referenced this pull request Jan 11, 2023
@kartg
Reintroducing backwards compatibility logic in certain classes
1abd307 
This reverts changes made in opensearch-project#4728 and opensearch-project#4702. These were only made in main and not backported to 2.x
This change also adds unit tests for IndexMetadataGenerations

Signed-off-by: Kartik Ganesh <gkart@amazon.com>
kartg added a commit to kartg/OpenSearch that referenced this pull request Jan 12, 2023
@kartg
Reintroducing backwards compatibility logic in certain classes
d6e4a2b 
This reverts changes made in opensearch-project#4728 and opensearch-project#4702. These were only made in main and not backported to 2.x
This change also adds unit tests for IndexMetadataGenerations

Signed-off-by: Kartik Ganesh <gkart@amazon.com>
kartg added a commit that referenced this pull request Jan 13, 2023
@kartg
Enhance searchable snapshots to enable a read-only view of older snap…
095c222 
…shots (#5812)

* Enhance searchable snapshots to enable a read-only view of older snapshots (#5429)

* Enhance searchable snapshots to enable a read-only view of older snapshots

This change removes the guardrails around N-1 backward compatibility and uses Lucene's "expert" APIs to read snapshots (Lucene segments) older than N-1 on a best-effort basis. The functionality is gated by an additional feature flag, separate from the searchable snapshots flag. Note that the Lucene integration is rather inefficient because the necessary "expert" Lucene APIs are still package-private.

Signed-off-by: Kartik Ganesh <gkart@amazon.com>

* Added some unit tests

This change also includes a test index ZIP file for the unit tests. The change also introduces a bug fix in the readAnySegmentsInfo method to close the reader before returning the SegmentInfos object - this avoids dangling/open file handles.

Signed-off-by: Kartik Ganesh <gkart@amazon.com>

* Incorporating PR feedback

Signed-off-by: Kartik Ganesh <gkart@amazon.com>

* Incorporate PR comments from andrross

Signed-off-by: Kartik Ganesh <gkart@amazon.com>

* Remove use of IndexSetting for minimum version for snapshots backwards compatibility

Signed-off-by: Kartik Ganesh <gkart@amazon.com>

* Moved ES 6.3.0 test data to a subdirectory

This change also includes an update to the file name to clarify that it is an ES index, and changing the associated markdown file name to just README.md. All tests that reference this ZIP file have corresponding changes to the path they reference.

Signed-off-by: Kartik Ganesh <gkart@amazon.com>

* Update unit tests to use try-with-resources

Signed-off-by: Kartik Ganesh <gkart@amazon.com>

* Added FeatureFlagSetter helper class

Also refactored unit test classes to use the helper class.

Signed-off-by: Kartik Ganesh <gkart@amazon.com>

* Incorporating PR feedback from @mch2

Signed-off-by: Kartik Ganesh <gkart@amazon.com>

* Fix IndexSettingsTests

Updated the asserts in IndexSettingsTests to account for the new defaulting behavior.

Signed-off-by: Kartik Ganesh <gkart@amazon.com>

Signed-off-by: Kartik Ganesh <gkart@amazon.com>

* Fixed compile issues after cherry-pick

Note that the unit tests are still failing at this commit since the Lucene 9 libraries no longer hold constants for Lucene 7 and below, so the fromId logic resolves the Lucene version to 8.

Signed-off-by: Kartik Ganesh <gkart@amazon.com>

* Fix multiple aspects of version resolution

This change fixes resolution of the Lucene version for legacy versions since the Lucene 9 libraries no longer hold constants for Lucene 7 and below. The change also updates DECLARED_VERSIONS to derive from the Versions class rather than LegacyESVersions (thereby ignoring legacy versions). This in turn required a change to the minimumIndexCompatibleVersion logic for LegacyESVersion. Finally, the testMinimumIndexCompatibilityVersion unit test was updated to use accurate version identifiers.

All unit tests pass and the code compiles, but actual functionality is still broken because some backwards compatibility logic was removed in the current branch that is retained in 2.x

Signed-off-by: Kartik Ganesh <gkart@amazon.com>

* Reintroducing backwards compatibility logic in certain classes

This reverts changes made in #4728 and #4702. These were only made in main and not backported to 2.x
This change also adds unit tests for IndexMetadataGenerations

Signed-off-by: Kartik Ganesh <gkart@amazon.com>

* Declared LegacyESVersion constants for better readability

This commit also includes a correction to documentation, and removes the unnecessary "afterWriteSnapBlob" runnable from BlobStoreRepository

Signed-off-by: Kartik Ganesh <gkart@amazon.com>

* Fixing PR reference in CHANGELOG

Signed-off-by: Kartik Ganesh <gkart@amazon.com>

* Revert CHANGELOG update

This feature was released in 2.5.0 so it no longer needs to be listed in the changelog.

Signed-off-by: Kartik Ganesh <gkart@amazon.com>

* Remove outdated comment

Signed-off-by: Kartik Ganesh <gkart@amazon.com>

Signed-off-by: Kartik Ganesh <gkart@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@reta reta reta approved these changes

Assignees
No one assigned
Labels
non-issue bugs / unexpected behaviors that end up non issues; audit trail simple changes that aren't issues >upgrade Label used when upgrading library dependencies (e.g., Lucene) v3.0.0 Issues and PRs related to version 3.0.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@nknize @reta