[BUG] Total hits value behavior change in the response when using search_after in paginated search requests #9013

tomchlee · 2023-08-01T01:22:15Z

Describe the bug

Recently we upgraded to Opensearch 2.8.0 from 2.6.0 and noticed that a behavior change in the total hits value in the response when using search_after in paginated search requests. In this case, there were no ongoing indexing activities (same search requests would return the same number of documents/total hits). Once we started using search_after parameter in the search request to page through the results and came to the very last page result, if we make another search_after request using the sort value, the response returned with no documents and total hits value of 1, whereas the total hits in previous search_after responses all had the same value as in the initial search response. In 2.6.0, total hits value remained the same even for search_after request using the sort value of the very last page result (and no documents).

We also tried upgrading to 2.9.0 and performed the same test, and found the total hits value was not only different from the initial search_after request but also different for each search_after request with the total hits value decreasing for each subsequent request.

Is this behavior an intended change? Keep in mind, no new documents were indexed during the test.

To Reproduce

See description.

Expected behavior

Total hits value remained the same even for search_after request using the sort value of the very last page result (and no documents).

Plugins
Please list all plugins currently enabled.

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

OS: Linux
Version [e.g. 22]

Additional context
Add any other context about the problem here.

dblock · 2023-08-03T15:40:06Z

AFAIK this is not expected, do you have a simple repro? care to post it here, and maybe try to turn it into a YAML REST test?

tomchlee · 2023-08-04T21:49:20Z

@dblock To reproduce:

Setup two clusters, cluster-1 and cluster-2, and configure cluster-2 as remote cluster to cluster-1.
For each cluster, create the following index template:

PUT _index_template/my-test
{
  "index_patterns": [
    "my-test"
  ],
  "template": {
    "settings": {
      "number_of_shards": "2",
      "number_of_replicas": "1",
      "routing_partition_size": "1"
    },
    "mappings": {
      "dynamic": "false",
      "_routing": {
        "required": true
      },
      "properties": {
        "foo": {
          "type": "keyword",
          "ignore_above": 32766
        },
        "ts": {
          "index": true,
          "doc_values": true,
          "type": "date"
        }
      }
    }
  }
}

For each cluster, index 3 documents such as below:

POST my-test/_doc?routing=bar
{
  "foo": "bar",
  "ts": "2023-07-31T22:30:37.038503154Z"
}

Send initial cross-cluster search request to cluster-1 and note the total hits value in the response:

POST my-test,cluster-2:my-test/_search
{
  "size": 4,
  "query": {
    "bool": {
      "must": [
        {
          "term": {
            "foo": "bar"
          }
        }
      ]
    }
  },
  "search_after": [
    1690835437038,
          "013jwYkBSuTL3GX2hT8X"
  ],
  "sort": [
    {
      "ts": "desc"
    },
    {
      "_id": "desc"
    }
  ]
}

Send the same cross-cluster search request with search_after parameter (using the sort array values from the last document in the returned list in the previous response) to cluster-1 and note the total hits value in the response.
Repeat the previous step.

After running these steps, total hits value for the three search requests were:

v2.6.0: 6, 6, 6
v2.8.0: 6, 6, 3

dblock · 2023-08-09T22:51:40Z

Just to confirm, this is not happening in a single node setup (not cross cluster)?

tomchlee · 2023-08-09T23:10:07Z

We test with cross cluster as that was the setup in our environment. i just tried with single cluster with 6 documents and was able to reproduce. Also, this only seems to happen if routing is used.

msfroh · 2023-08-16T16:33:17Z

Maybe related to #6596 ?

gashutos · 2023-08-25T07:18:48Z

I am not able to repro this on single node cluster.

Below steps I tried.

DELETE my-test

PUT _index_template/my-test
{
  "index_patterns": [
    "my-test"
  ],
  "template": {
    "settings": {
      "number_of_shards": "2",
      "number_of_replicas": "1",
      "routing_partition_size": "1"
    },
    "mappings": {
      "dynamic": "false",
      "_routing": {
        "required": true
      },
      "properties": {
        "foo": {
          "type": "keyword",
          "ignore_above": 32766
        },
        "ts": {
          "index": true,
          "doc_values": true,
          "type": "date"
        }
      }
    }
  }
}

POST my-test/_doc?routing=bar
{
  "foo": "bar",
  "ts": "2023-07-31T22:30:31.038503154Z"
}

POST my-test/_doc?routing=bar
{
  "foo": "bar",
  "ts": "2023-07-31T22:30:32.038503154Z"
}

POST my-test/_doc?routing=bar
{
  "foo": "bar",
  "ts": "2023-07-31T22:30:33.038503154Z"
}

POST my-test/_doc?routing=bar
{
  "foo": "bar",
  "ts": "2023-07-31T22:30:34.038503154Z"
}

POST my-test/_doc?routing=bar
{
  "foo": "bar",
  "ts": "2023-07-31T22:30:35.038503154Z"
}

POST my-test/_doc?routing=bar
{
  "foo": "bar",
  "ts": "2023-07-31T22:30:36.038503154Z"
}

POST my-test/_search
{
  "size": 4,
  "query": {
    "bool": {
      "must": [
        {
          "term": {
            "foo": "bar"
          }
        }
      ]
    }
  },
  "sort": [
    {
      "ts": "desc"
    },
    {
      "_id": "desc"
    }
  ]
}

POST my-test/_search
{
  "size": 4,
  "query": {
    "bool": {
      "must": [
        {
          "term": {
            "foo": "bar"
          }
        }
      ]
    }
  },
  "search_after": [
    1690842634038,
          "OiyLK4oB9ZmHRf3YM7JJ"
  ],
  "sort": [
    {
      "ts": "desc"
    },
    {
      "_id": "desc"
    }
  ]
}

POST my-test/_search
{
  "size": 4,
  "query": {
    "bool": {
      "must": [
        {
          "term": {
            "foo": "bar"
          }
        }
      ]
    }
  },
  "search_after": [
    1690842631038,
          "OyyLK4oB9ZmHRf3YT7Kw"
  ],
  "sort": [
    {
      "ts": "desc"
    },
    {
      "_id": "desc"
    }
  ]
}

Output

# DELETE my-test
{
  "acknowledged": true
}
# PUT _index_template/my-test
{
  "acknowledged": true
}
# POST my-test/_doc?routing=bar
{
  "_index": "my-test",
  "_id": "67eTK4oBEJJCFQ1y2oE1",
  "_version": 1,
  "result": "created",
  "_shards": {
    "total": 2,
    "successful": 1,
    "failed": 0
  },
  "_seq_no": 0,
  "_primary_term": 1
}
# POST my-test/_doc?routing=bar
{
  "_index": "my-test",
  "_id": "QCyTK4oB9ZmHRf3Y2rLd",
  "_version": 1,
  "result": "created",
  "_shards": {
    "total": 2,
    "successful": 2,
    "failed": 0
  },
  "_seq_no": 1,
  "_primary_term": 1
}
# POST my-test/_doc?routing=bar
{
  "_index": "my-test",
  "_id": "BsyTK4oBI7gOl26J2-SF",
  "_version": 1,
  "result": "created",
  "_shards": {
    "total": 2,
    "successful": 2,
    "failed": 0
  },
  "_seq_no": 2,
  "_primary_term": 1
}
# POST my-test/_doc?routing=bar
{
  "_index": "my-test",
  "_id": "7LeTK4oBEJJCFQ1y3IEq",
  "_version": 1,
  "result": "created",
  "_shards": {
    "total": 2,
    "successful": 2,
    "failed": 0
  },
  "_seq_no": 3,
  "_primary_term": 1
}
# POST my-test/_doc?routing=bar
{
  "_index": "my-test",
  "_id": "QSyTK4oB9ZmHRf3Y3LLQ",
  "_version": 1,
  "result": "created",
  "_shards": {
    "total": 2,
    "successful": 2,
    "failed": 0
  },
  "_seq_no": 4,
  "_primary_term": 1
}
# POST my-test/_doc?routing=bar
{
  "_index": "my-test",
  "_id": "B8yTK4oBI7gOl26J3eR3",
  "_version": 1,
  "result": "created",
  "_shards": {
    "total": 2,
    "successful": 2,
    "failed": 0
  },
  "_seq_no": 5,
  "_primary_term": 1
}
# POST my-test/_search
{
  "took": 5,
  "timed_out": false,
  "_shards": {
    "total": 2,
    "successful": 2,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 6,
      "relation": "eq"
    },
    "max_score": null,
    "hits": [
      {
        "_index": "my-test",
        "_id": "B8yTK4oBI7gOl26J3eR3",
        "_score": null,
        "_routing": "bar",
        "_source": {
          "foo": "bar",
          "ts": "2023-07-31T22:30:36.038503154Z"
        },
        "sort": [
          1690842636038,
          "B8yTK4oBI7gOl26J3eR3"
        ]
      },
      {
        "_index": "my-test",
        "_id": "QSyTK4oB9ZmHRf3Y3LLQ",
        "_score": null,
        "_routing": "bar",
        "_source": {
          "foo": "bar",
          "ts": "2023-07-31T22:30:35.038503154Z"
        },
        "sort": [
          1690842635038,
          "QSyTK4oB9ZmHRf3Y3LLQ"
        ]
      },
      {
        "_index": "my-test",
        "_id": "7LeTK4oBEJJCFQ1y3IEq",
        "_score": null,
        "_routing": "bar",
        "_source": {
          "foo": "bar",
          "ts": "2023-07-31T22:30:34.038503154Z"
        },
        "sort": [
          1690842634038,
          "7LeTK4oBEJJCFQ1y3IEq"
        ]
      },
      {
        "_index": "my-test",
        "_id": "BsyTK4oBI7gOl26J2-SF",
        "_score": null,
        "_routing": "bar",
        "_source": {
          "foo": "bar",
          "ts": "2023-07-31T22:30:33.038503154Z"
        },
        "sort": [
          1690842633038,
          "BsyTK4oBI7gOl26J2-SF"
        ]
      }
    ]
  }
}
# POST my-test/_search
{
  "took": 6,
  "timed_out": false,
  "_shards": {
    "total": 2,
    "successful": 2,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 6,
      "relation": "eq"
    },
    "max_score": null,
    "hits": [
      {
        "_index": "my-test",
        "_id": "7LeTK4oBEJJCFQ1y3IEq",
        "_score": null,
        "_routing": "bar",
        "_source": {
          "foo": "bar",
          "ts": "2023-07-31T22:30:34.038503154Z"
        },
        "sort": [
          1690842634038,
          "7LeTK4oBEJJCFQ1y3IEq"
        ]
      },
      {
        "_index": "my-test",
        "_id": "BsyTK4oBI7gOl26J2-SF",
        "_score": null,
        "_routing": "bar",
        "_source": {
          "foo": "bar",
          "ts": "2023-07-31T22:30:33.038503154Z"
        },
        "sort": [
          1690842633038,
          "BsyTK4oBI7gOl26J2-SF"
        ]
      },
      {
        "_index": "my-test",
        "_id": "QCyTK4oB9ZmHRf3Y2rLd",
        "_score": null,
        "_routing": "bar",
        "_source": {
          "foo": "bar",
          "ts": "2023-07-31T22:30:32.038503154Z"
        },
        "sort": [
          1690842632038,
          "QCyTK4oB9ZmHRf3Y2rLd"
        ]
      },
      {
        "_index": "my-test",
        "_id": "67eTK4oBEJJCFQ1y2oE1",
        "_score": null,
        "_routing": "bar",
        "_source": {
          "foo": "bar",
          "ts": "2023-07-31T22:30:31.038503154Z"
        },
        "sort": [
          1690842631038,
          "67eTK4oBEJJCFQ1y2oE1"
        ]
      }
    ]
  }
}
# POST my-test/_search
{
  "took": 4,
  "timed_out": false,
  "_shards": {
    "total": 2,
    "successful": 2,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 6,
      "relation": "eq"
    },
    "max_score": null,
    "hits": [
      {
        "_index": "my-test",
        "_id": "67eTK4oBEJJCFQ1y2oE1",
        "_score": null,
        "_routing": "bar",
        "_source": {
          "foo": "bar",
          "ts": "2023-07-31T22:30:31.038503154Z"
        },
        "sort": [
          1690842631038,
          "67eTK4oBEJJCFQ1y2oE1"
        ]
      }
    ]
  }
}

gashutos · 2023-08-25T07:19:26Z

@tomchlee Could you help me verify if I am reproing this in right steps ?

dblock · 2023-08-25T13:20:47Z

@gashutos Are you running your tests on 2.8 or 2.9?

tomchlee · 2023-08-25T16:51:39Z

@gashutos your steps were mostly correct except for the search_after value for 2nd and 3rd searches; the search_after value should match the sort value in the last document in the returned hits array from the previous search response, so 2nd search (the first with search_after) should be like this:

POST my-test/_search
{
  "size": 4,
  "query": {
    "bool": {
      "must": [
        {
          "term": {
            "foo": "bar"
          }
        }
      ]
    }
  },
  "search_after": [
    1690842633038,
     "BsyTK4oBI7gOl26J2-SF"
  ],
  "sort": [
    {
      "ts": "desc"
    },
    {
      "_id": "desc"
    }
  ]
}

and 3rd search should be:

POST my-test/_search
{
  "size": 4,
  "query": {
    "bool": {
      "must": [
        {
          "term": {
            "foo": "bar"
          }
        }
      ]
    }
  },
  "search_after": [
       1690842631038,
       "67eTK4oBEJJCFQ1y2oE1"
  ],
  "sort": [
    {
      "ts": "desc"
    },
    {
      "_id": "desc"
    }
  ]
}

Number of documents returned by the three search requests should be: 4, 2, 0.

gashutos · 2023-08-27T20:21:10Z

No luck on reproduction yet ! Tested on 2.8.0.
@tomchlee can you paste your output on single-node cluster ?

tomchlee · 2023-08-28T03:17:23Z

@gashutos i didn't try it with single-node cluster; i tested with a single cluster with 2 data nodes as well as 2 clusters each with 2 data nodes.

gashutos · 2023-08-28T03:29:55Z

@tomchlee fine, output of 2 node cluster will also help.

tomchlee · 2023-08-28T16:45:45Z

@gashutos
Cluster setup: 1 master, 1 coordinating and 2 data nodes.

Output (note hits.total.value is different in the last search response):

# DELETE my-test
{
  "acknowledged": true
}
# PUT _index_template/my-test
{
  "acknowledged": true
}
# POST my-test/_doc?routing=bar
{ 
  "_index": "my-test",
  "_id": "UkLpPIoBtuDhdmzKy-hG",
  "_version": 1,
  "result": "created",
  "_shards": {
    "total": 2,
    "successful": 1,
    "failed": 0
  },
  "_seq_no": 0,
  "_primary_term": 1
}
# POST my-test/_doc?routing=bar
{
  "_index": "my-test",
  "_id": "U0LxPIoBtuDhdmzKgeih",
  "_version": 1,
  "result": "created",
  "_shards": { 
    "total": 2,
    "successful": 2,
    "failed": 0
  },
  "_seq_no": 1,
  "_primary_term": 1
}
# POST my-test/_doc?routing=bar
{ 
  "_index": "my-test",
  "_id": "VELzPIoBtuDhdmzKZ-h-",
  "_version": 1,
  "result": "created",
  "_shards": { 
    "total": 2,
    "successful": 2,
    "failed": 0
  },
  "_seq_no": 2,
  "_primary_term": 1
}
# POST my-test/_doc?routing=bar
{ 
  "_index": "my-test",
  "_id": "VUL0PIoBtuDhdmzKVegE",
  "_version": 1,
  "result": "created",
  "_shards": { 
    "total": 2,
    "successful": 2,
    "failed": 0
  },
  "_seq_no": 3,
  "_primary_term": 1
}
# POST my-test/_doc?routing=bar
{  
  "_index": "my-test",
  "_id": "VkL1PIoBtuDhdmzKF-i9",
  "_version": 1,
  "result": "created",
  "_shards": { 
    "total": 2,
    "successful": 2,
    "failed": 0
  },
  "_seq_no": 4,
  "_primary_term": 1
}
# POST my-test/_doc?routing=bar
{  
  "_index": "my-test",
  "_id": "V0L1PIoBtuDhdmzK8eil",
  "_version": 1,
  "result": "created",
  "_shards": {  
    "total": 2,
    "successful": 2,
    "failed": 0
  },
  "_seq_no": 5,
  "_primary_term": 1
}
# POST my-test/_search
{  
  "took": 22,
  "timed_out": false,
  "_shards": { 
    "total": 2,
    "successful": 2,
    "skipped": 0,
    "failed": 0
  },
  "hits": { 
    "total": {  
      "value": 6,
      "relation": "eq"
    },
    "max_score": null,
    "hits": [ 
      { 
        "_index": "my-test",
        "_id": "V0L1PIoBtuDhdmzK8eil",
        "_score": null,
        "_routing": "bar",
        "_source": { 
          "foo": "bar",
          "ts": "2023-07-31T22:30:36.038503154Z"
        },
        "sort": [ 
          1690842636038,
          "V0L1PIoBtuDhdmzK8eil"
        ]
      },
      { 
        "_index": "my-test",
        "_id": "VkL1PIoBtuDhdmzKF-i9",
        "_score": null,
        "_routing": "bar",
        "_source": { 
          "foo": "bar",
          "ts": "2023-07-31T22:30:35.038503154Z"
        },
        "sort": [  
          1690842635038,
          "VkL1PIoBtuDhdmzKF-i9"
        ]
      },
      {  
        "_index": "my-test",
        "_id": "VUL0PIoBtuDhdmzKVegE",
        "_score": null,
        "_routing": "bar",
        "_source": {  
          "foo": "bar",
          "ts": "2023-07-31T22:30:34.038503154Z"
        },
        "sort": [ 
          1690842634038,
          "VUL0PIoBtuDhdmzKVegE"
        ]
      },
      { 
        "_index": "my-test",
        "_id": "VELzPIoBtuDhdmzKZ-h-",
        "_score": null,
        "_routing": "bar",
        "_source": { 
          "foo": "bar",
          "ts": "2023-07-31T22:30:33.038503154Z"
        },
        "sort": [ 
          1690842633038,
          "VELzPIoBtuDhdmzKZ-h-"
        ]
      }
    ]
  }
}
# POST my-test/_search
{ 
  "took": 5,
  "timed_out": false,
  "_shards": {  
    "total": 2,
    "successful": 2,
    "skipped": 0,
    "failed": 0
  },
  "hits": {  
    "total": {  
      "value": 6,
      "relation": "eq"
    },
    "max_score": null,
    "hits": [  
      { 
        "_index": "my-test",
        "_id": "U0LxPIoBtuDhdmzKgeih",
        "_score": null,
        "_routing": "bar",
        "_source": { 
          "foo": "bar",
          "ts": "2023-07-31T22:30:32.038503154Z"
        },
        "sort": [ 
          1690842632038,
          "U0LxPIoBtuDhdmzKgeih"
        ]
      },
      { 
        "_index": "my-test",
        "_id": "UkLpPIoBtuDhdmzKy-hG",
        "_score": null,
        "_routing": "bar",
        "_source": { 
          "foo": "bar",
          "ts": "2023-07-31T22:30:31.038503154Z"
        },
        "sort": [  
          1690842631038,
          "UkLpPIoBtuDhdmzKy-hG"
        ]
      }
    ]
  }
}
# POST my-test/_search
{ 
  "took": 5,
  "timed_out": false,
  "_shards": { 
    "total": 2,
    "successful": 2,
    "skipped": 0,
    "failed": 0
  },
  "hits": {  
    "total": { 
      "value": 1,
      "relation": "eq"
    },
    "max_score": null,
    "hits": [ 

    ]
  }
}

gashutos · 2023-08-28T19:51:54Z

Thanks @tomchlee. At least on 2 node cluster (both data), this is not reproducing on both 2.8.0 & 2.9.0.
Let me set up exact same cofigurations as yours and see.

gashutos · 2023-09-01T06:29:47Z

#9683 , raised PR for fixing this.

gashutos · 2023-09-05T16:04:07Z

@tomchlee would you mind sharing your usecase here ? Like, what purpose you are using hits.total.value in paginated search_after queries ?
We are thinking to change this behavior to honor search_after value to calculate hits.total.value to give user hint where they are in pagination. #9717

tomchlee · 2023-09-06T00:30:32Z

@gashutos our stateless frontend applications always display total search result counts via hits.total.value while going through paginated search_after queries, so if this behavior were to change, the applications would have to cache the value from the original query without search_after.

reta · 2023-09-06T01:03:57Z

@gashutos I really like your idea of enhancing the search_after by providing the positional context, may be could keep the hits.total.value as is, but include new hits.search_after section with before and after values for hits?

tomchlee added bug Something isn't working untriaged labels Aug 1, 2023

anasalkouz added the Search Search query, autocomplete ...etc label Aug 11, 2023

github-project-automation bot added this to Search Project Board Aug 11, 2023

github-project-automation bot moved this to 🆕 New in Search Project Board Aug 11, 2023

anasalkouz added untriaged and removed untriaged labels Aug 11, 2023

msfroh removed the untriaged label Aug 16, 2023

dblock mentioned this issue Aug 24, 2023

[BUG] OpenSearch non-deterministically returning fewer than size hits when search_after is configured even if hits exist #9537

Closed

gashutos mentioned this issue Sep 1, 2023

Disable shard/segment level search_after short cutting if track_total_hits != false #9683

Merged

6 tasks

reta closed this as completed in #9683 Sep 2, 2023

github-project-automation bot moved this from 🆕 New to ✅ Done in Search Project Board Sep 2, 2023

gashutos mentioned this issue Sep 5, 2023

[Feature Request] Paginated search queries should hint current position of page in individual page queries #9717

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] Total hits value behavior change in the response when using search_after in paginated search requests #9013

[BUG] Total hits value behavior change in the response when using search_after in paginated search requests #9013

tomchlee commented Aug 1, 2023

dblock commented Aug 3, 2023

tomchlee commented Aug 4, 2023 •

edited

Loading

dblock commented Aug 9, 2023

tomchlee commented Aug 9, 2023

msfroh commented Aug 16, 2023

gashutos commented Aug 25, 2023 •

edited

Loading

gashutos commented Aug 25, 2023

dblock commented Aug 25, 2023

tomchlee commented Aug 25, 2023

gashutos commented Aug 27, 2023

tomchlee commented Aug 28, 2023

gashutos commented Aug 28, 2023 •

edited

Loading

tomchlee commented Aug 28, 2023

gashutos commented Aug 28, 2023

gashutos commented Sep 1, 2023

gashutos commented Sep 5, 2023 •

edited

Loading

tomchlee commented Sep 6, 2023

reta commented Sep 6, 2023

[BUG] Total hits value behavior change in the response when using search_after in paginated search requests #9013

[BUG] Total hits value behavior change in the response when using search_after in paginated search requests #9013

Comments

tomchlee commented Aug 1, 2023

dblock commented Aug 3, 2023

tomchlee commented Aug 4, 2023 • edited Loading

dblock commented Aug 9, 2023

tomchlee commented Aug 9, 2023

msfroh commented Aug 16, 2023

gashutos commented Aug 25, 2023 • edited Loading

gashutos commented Aug 25, 2023

dblock commented Aug 25, 2023

tomchlee commented Aug 25, 2023

gashutos commented Aug 27, 2023

tomchlee commented Aug 28, 2023

gashutos commented Aug 28, 2023 • edited Loading

tomchlee commented Aug 28, 2023

gashutos commented Aug 28, 2023

gashutos commented Sep 1, 2023

gashutos commented Sep 5, 2023 • edited Loading

tomchlee commented Sep 6, 2023

reta commented Sep 6, 2023

tomchlee commented Aug 4, 2023 •

edited

Loading

gashutos commented Aug 25, 2023 •

edited

Loading

gashutos commented Aug 28, 2023 •

edited

Loading

gashutos commented Sep 5, 2023 •

edited

Loading