Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Backport 2.x][Search Pipelines] Support ad hoc pipelines #7253 #7473

Merged

Conversation

msfroh
Copy link
Collaborator

@msfroh msfroh commented May 9, 2023

Description

Backport #7253 to 2.x (since auto-backport failed).

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

msfroh added 2 commits May 9, 2023 00:40
* [Search Pipelines] Support ad hoc pipelines

This change allows a search pipeline to be defined within a search
request body. This will take precedence over any other search pipeline,
and allows ad hoc testing of pipeline configurations before persisting
the pipeline definition in cluster state.

Signed-off-by: Michael Froh <froh@amazon.com>

* Added changelog entry

Signed-off-by: Michael Froh <froh@amazon.com>

* Incorporate feedback from @reta

Resolve + transform request into PipelinedRequest, then use that to
transform the search response.

Signed-off-by: Michael Froh <froh@amazon.com>

* Incorporate more feedback from @reta

1. Create all streams in try-with-resources.
2. Keep methods in Pipeline package-private.
3. Remove spurious character in CHANGELOG.

Also, I remembered that in a future change, I'm going to call the no-op
pipeline "_none" (like when bypassing an ingest pipeline) so I should
name it "_none" from the start.

Signed-off-by: Michael Froh <froh@amazon.com>

* Incorporate feedback from @andrross

- Added opensearch.internal annotation to PipelinedRequest.
- Made PipelinedRequest final.
- Removed TODO from PipelinedRequest.
- Moved changelog entry to 2.x.

Signed-off-by: Michael Froh <froh@amazon.com>

---------

Signed-off-by: Michael Froh <froh@amazon.com>
(cherry picked from commit c43d713)
Signed-off-by: Michael Froh <froh@amazon.com>
@github-actions
Copy link
Contributor

github-actions bot commented May 9, 2023

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

github-actions bot commented May 9, 2023

Gradle Check (Jenkins) Run Completed with:

  • RESULT: UNSTABLE ❕
  • TEST FAILURES:
      1 org.opensearch.indices.replication.SegmentReplicationIT.testScrollWithOngoingSegmentReplication
      1 org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.testIndexCreateBlockIsRemovedWhenAnyNodesNotExceedHighWatermarkWithAutoReleaseEnabled

@codecov-commenter
Copy link

Codecov Report

Merging #7473 (197d0ed) into 2.x (52630c4) will decrease coverage by 0.07%.
The diff coverage is 42.10%.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

@@             Coverage Diff              @@
##                2.x    #7473      +/-   ##
============================================
- Coverage     70.34%   70.27%   -0.07%     
+ Complexity    59895    59868      -27     
============================================
  Files          4873     4876       +3     
  Lines        288170   288271     +101     
  Branches      41897    41911      +14     
============================================
- Hits         202699   202576     -123     
- Misses        68470    68665     +195     
- Partials      17001    17030      +29     
Impacted Files Coverage Δ
...arch/gradle/testclusters/ExtensionsProperties.java 0.00% <0.00%> (ø)
...nsearch/gradle/testclusters/OpenSearchCluster.java 0.00% <0.00%> (ø)
...opensearch/gradle/testclusters/OpenSearchNode.java 0.00% <0.00%> (ø)
.../gradle/testclusters/TestClusterConfiguration.java 0.00% <ø> (ø)
...search/gradle/testclusters/TestExtensionsList.java 0.00% <0.00%> (ø)
...pensearch/search/RandomSearchRequestGenerator.java 94.87% <0.00%> (-0.99%) ⬇️
...opensearch/search/builder/SearchSourceBuilder.java 68.61% <60.00%> (+1.77%) ⬆️
...nsearch/search/pipeline/SearchPipelineService.java 83.15% <80.95%> (+0.54%) ⬆️
...pensearch/action/search/TransportSearchAction.java 70.99% <100.00%> (+1.58%) ⬆️
.../java/org/opensearch/search/pipeline/Pipeline.java 84.61% <100.00%> (+3.13%) ⬆️
... and 1 more

... and 476 files with indirect coverage changes

@kotwanikunal
Copy link
Member

@reta / @msfroh Can you please mark the conversations as resolved before we can merge it in?

@msfroh
Copy link
Collaborator Author

msfroh commented May 11, 2023

Can you please mark the conversations as resolved before we can merge it in?

@kotwanikunal -- done!

@kotwanikunal kotwanikunal merged commit 5280097 into opensearch-project:2.x May 11, 2023
@msfroh msfroh deleted the backport/backport-7253-to-2.x branch May 15, 2023 21:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

4 participants