Disable shard/segment level search_after short cutting if track_total_hits != false #9683

gashutos · 2023-09-01T06:28:11Z

Description

Context

Any _search query return hits.total.value in response like mentioned in below example. Which specifies how many hits are matching actual query (sort/search_after are not part of query).

i.e, check response for below query

GET logs-*/_search
{
        "query": {
          "match_all": {}
        },
        "sort" : [
          {"@timestamp" : "asc"}
        ],
        "search_after": ["1998-06-01"]
}

Response [partial]

{
  "took": 25,
  "timed_out": false,
  "_shards": {
    "total": 35,
    "successful": 35,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 10000,
      "relation": "gte"
    },
    "max_score": null,
    "hits": [
      {
        "_ind

This hits.total.value is capped at 10000 max as lucene can match at max 10000 documents in results.

If you want exact match in hits.total.value, we need to add track_total_hits=true in request parameter.
i.e.

GET logs-*/_search?track_total_hits=true
{
        "query": {
          "match_all": {}
        },
        "sort" : [
          {"@timestamp" : "asc"}
        ],
        "search_after": ["1998-06-01"]
}

Response for above query [partial]

{
  "took": 32,
  "timed_out": false,
  "_shards": {
    "total": 35,
    "successful": 35,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 247249096,
      "relation": "eq"
    },
    "max_score": null,
    "hits": [

If you check, it has exact matching value 247249096 in hits.total.value.

Above calculation results on additional computational costs on _search query to track total matching hits. So for performance sensitive queries, it is recommended to provide track_total_hits=falsle.
i.e

GET logs-*/_search?track_total_hits=false
{
        "query": {
          "match_all": {}
        },
        "sort" : [
          {"@timestamp" : "asc"}
        ],
        "search_after": ["1998-06-01"]
}

Response from above search query [partial]

{
  "took": 14,
  "timed_out": false,
  "_shards": {
    "total": 35,
    "successful": 35,
    "skipped": 25,
    "failed": 0
  },
  "hits": {
    "max_score": null,

Default behaviour for _search queries if track_total_hits is not provided as request input, is track matching hits up to 10000 documents as mentioned above.

Issue

Now with introduction of shard/segment level shortcutting [#6596] for search_after queries, it will skip issuing _search request for those shards/segments. So we will end up NOT tracking those documents hits which are matching query for that skipped shards/segmetns because of search_after based skipping.
This was caught in this issue. #9013.

Related Issues

Resolves #9013

Resolution in this PR

Use search_after shortcutting only if track_total_hits=false provided.

Additional note

There is no latency impact on _search queries even if we dont use shard/segment level skipping logic, now that we have this optimization available in Lucene itself with apache/lucene#12334.
It will end up issuing network call for search request for that shard but lucene query will return immediately since no matching documents and there wont be any fetch phase.
yeah I agree it will end up issuing additional network calls from coordinator to shard replica hosted data nodes.

I tested with 3 nodes cluster http_logs with 5 indices with 5 shards each. Latencies are almost similar. So no side -effect on performance with disabling this shortcutting if track_total_hits is not false.
This is P50, units are in ms.
No difference for P90, p99 as well.

Data Node Type.	Sort Mode	With track_total_hits != false	track_total_hits = false
r6g.2xlarge	asc with after	41	39
r6g.2xlarge	desc with after	103	101

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

gashutos · 2023-09-01T06:32:51Z

@dblock @backslasht @reta, when you get chance.
Hopefuly last bug of search_after.

I dont understand why hits.total.value is not considering search_after. I raised the PR to keep behaviour same, but it does not make sense to return matching hits same in all paginated responses.
Also it is capped @10000.

Anyway, lets make behaviour same as it used to be before, as we dont take any latency hit after lucene fix as mentioned

github-actions · 2023-09-01T06:38:35Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/24077/
CommitID: dd6967b
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-09-01T06:59:09Z

Compatibility status:

Checks if related components are compatible with change dd6967b

Incompatible components

Incompatible components: [https://github.com/opensearch-project/cross-cluster-replication.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

github-actions · 2023-09-01T16:49:58Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/24123/
CommitID: c127758
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-09-01T16:53:10Z

Compatibility status:

Checks if related components are compatible with change c127758

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

github-actions · 2023-09-01T17:33:11Z

Compatibility status:

Checks if related components are compatible with change 1f16245

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

github-actions · 2023-09-01T17:46:12Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/24137/
CommitID: 1f16245
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

…_hits != false Signed-off-by: gashutos <gashutos@amazon.com>

github-actions · 2023-09-01T18:29:19Z

Compatibility status:

Checks if related components are compatible with change 21a3679

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/reporting.git]

github-actions · 2023-09-01T18:32:57Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/24146/
CommitID: 21a3679
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Chaitanya Gohel <104654647+gashutos@users.noreply.github.com>

github-actions · 2023-09-01T19:25:48Z

Compatibility status:

Checks if related components are compatible with change eb0a9d5

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

github-actions · 2023-09-01T19:30:26Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/24153/
CommitID: eb0a9d5
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-09-01T20:24:02Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.indices.replication.SegmentReplicationRelocationIT.testPrimaryRelocation

URL: https://build.ci.opensearch.org/job/gradle-check/24158/
CommitID: eb0a9d5
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

backslasht

We could try to optimize for track_total_hits set to 1000 (default value) as well. But, that is a topic for separate PR. This is good to go.

opensearch-trigger-bot · 2023-09-02T18:56:26Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-9683-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 e98ded62d4b672a03acb4beb02bd047d7d4e1c69
# Push it to GitHub
git push --set-upstream origin backport/backport-9683-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-9683-to-2.x.

reta · 2023-09-02T20:03:37Z

@gashutos could you please do manual backport to 2.x? than you

…_hits != false (opensearch-project#9683) Signed-off-by: gashutos <gashutos@amazon.com> Signed-off-by: Chaitanya Gohel <104654647+gashutos@users.noreply.github.com>

gashutos · 2023-09-03T07:13:52Z

Thanks for reviews !
@backslasht I opened feature requst #9717
If we can honor search_after while tracking total hits, it will be usuful for long paginated searches.

…_hits != false (#9683) (#9716) Signed-off-by: gashutos <gashutos@amazon.com> Signed-off-by: Chaitanya Gohel <104654647+gashutos@users.noreply.github.com>

…_hits != false (opensearch-project#9683) Signed-off-by: gashutos <gashutos@amazon.com> Signed-off-by: Chaitanya Gohel <104654647+gashutos@users.noreply.github.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com>

…_hits != false (opensearch-project#9683) Signed-off-by: gashutos <gashutos@amazon.com> Signed-off-by: Chaitanya Gohel <104654647+gashutos@users.noreply.github.com> Signed-off-by: Ivan Brusic <ivan.brusic@flocksafety.com>

happyjoblzp · 2024-03-29T04:01:00Z

server/src/main/java/org/opensearch/search/SearchService.java

+            && minMax != null
+            && primarySortField != null
+            && primarySortField.missing() == null
+            && Objects.equals(trackTotalHitsUpto, SearchContext.TRACK_TOTAL_HITS_DISABLED)) {


@gashutos
Hello, I have a question. In these two scenarios：trackTotalHitsUpto=null and trackTotalHitsUpto < DEFAULT_TRACK_TOTAL_HITS_UP_TO，Is it possible to go through the optimization of shard/segment level shortcutting？

trackTotalHitsUpto=null
This should trigger optimization.
trackTotalHitsUpto < DEFAULT_TRACK_TOTAL_HITS_UP_TO
This should not.

However, We have made a fix in Lucene itself for this optimizations.
apache/lucene#12334

With this lucene optimization, even if we have this OpenSearch shortcutting disabled, it wont spend much time in actual lucene logic and it would be up par with shortcutting.

@gashutos Tks，there may be another problem for query with "search_after": [null, 1000000]; searchAfterPrimary equal null, minMax.compareMin will throw NPE, this behaviour will not same as before

example:

curl -XGET "127.0.0.1:9200/test/_search?pretty&track_total_hits=false" -H'Content-Type:application/json' -d' { "query": { "match_all": {} }, "sort":[{"name":{"order":"asc"}}, {"Id":{"order":"desc"}}], "search_after": [null, 1000000] }' { "error" : { "root_cause" : [ { "type" : "null_pointer_exception", "reason" : null } ], "type" : "search_phase_execution_exception", "reason" : "all shards failed", "phase" : "can_match", "grouped" : true, "failed_shards" : [ { "shard" : 112820878, "index" : "search_qingzhi", "node" : "127.0.0.1,51651,1711676797164", "reason" : { "type" : "null_pointer_exception", "reason" : null } } ], "caused_by" : { "type" : "null_pointer_exception", "reason" : null, "caused_by" : { "type" : "null_pointer_exception", "reason" : null } } }, "status" : 500 }

…_hits != false (opensearch-project#9683) Signed-off-by: gashutos <gashutos@amazon.com> Signed-off-by: Chaitanya Gohel <104654647+gashutos@users.noreply.github.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com>

gashutos added the skip-changelog label Sep 1, 2023

gashutos mentioned this pull request Sep 1, 2023

[BUG] Total hits value behavior change in the response when using search_after in paginated search requests #9013

Closed

gashutos force-pushed the main branch from dd6967b to 81adf2a Compare September 1, 2023 07:30

gashutos force-pushed the main branch from c127758 to 1f16245 Compare September 1, 2023 17:02

gashutos mentioned this pull request Sep 1, 2023

[Meta] Segment Replication flaky test failures #8279

Closed

16 tasks

Disable shard/segment level search_after short cutting if track_total…

21a3679

…_hits != false Signed-off-by: gashutos <gashutos@amazon.com>

gashutos force-pushed the main branch from 1f16245 to 21a3679 Compare September 1, 2023 17:59

Merge branch 'main' into main

eb0a9d5

Signed-off-by: Chaitanya Gohel <104654647+gashutos@users.noreply.github.com>

backslasht approved these changes Sep 2, 2023

View reviewed changes

reta merged commit e98ded6 into opensearch-project:main Sep 2, 2023

reta added the backport 2.x Backport to 2.x branch label Sep 2, 2023

opensearch-trigger-bot bot added the backport-failed label Sep 2, 2023

gashutos mentioned this pull request Sep 3, 2023

[Backport 2.x] Disable shard/segment level search_after short cutting if track_total_hits != false (#9683) #9716

Merged

gashutos mentioned this pull request Sep 5, 2023

[Feature Request] Paginated search queries should hint current position of page in individual page queries #9717

Open

happyjoblzp reviewed Mar 29, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Disable shard/segment level search_after short cutting if track_total_hits != false #9683

Disable shard/segment level search_after short cutting if track_total_hits != false #9683

gashutos commented Sep 1, 2023

gashutos commented Sep 1, 2023 •

edited

Loading

github-actions bot commented Sep 1, 2023

github-actions bot commented Sep 1, 2023

github-actions bot commented Sep 1, 2023

github-actions bot commented Sep 1, 2023

github-actions bot commented Sep 1, 2023

github-actions bot commented Sep 1, 2023

github-actions bot commented Sep 1, 2023

github-actions bot commented Sep 1, 2023

github-actions bot commented Sep 1, 2023

github-actions bot commented Sep 1, 2023

github-actions bot commented Sep 1, 2023

backslasht left a comment

opensearch-trigger-bot bot commented Sep 2, 2023

reta commented Sep 2, 2023

gashutos commented Sep 3, 2023 •

edited

Loading

happyjoblzp Mar 29, 2024

gashutos Mar 29, 2024

happyjoblzp Mar 29, 2024

Disable shard/segment level search_after short cutting if track_total_hits != false #9683

Disable shard/segment level search_after short cutting if track_total_hits != false #9683

Conversation

gashutos commented Sep 1, 2023

Description

Context

Issue

Related Issues

Resolution in this PR

Additional note

Check List

gashutos commented Sep 1, 2023 • edited Loading

github-actions bot commented Sep 1, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 1, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Sep 1, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 1, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Sep 1, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Sep 1, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 1, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Sep 1, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 1, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Sep 1, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 1, 2023

Gradle Check (Jenkins) Run Completed with:

backslasht left a comment

Choose a reason for hiding this comment

opensearch-trigger-bot bot commented Sep 2, 2023

reta commented Sep 2, 2023

gashutos commented Sep 3, 2023 • edited Loading

happyjoblzp Mar 29, 2024

Choose a reason for hiding this comment

gashutos Mar 29, 2024

Choose a reason for hiding this comment

happyjoblzp Mar 29, 2024

Choose a reason for hiding this comment

gashutos commented Sep 1, 2023 •

edited

Loading

gashutos commented Sep 3, 2023 •

edited

Loading