multi_match lucene query broken since 8.7.0

[neominik]

Elasticsearch Version

Version: 8.7.1, Build: docker/f229ed3f893a515d590d0f39b05f68913e2d9b53/2023-04-27T04:33:42.127815583Z, JVM: 20.0.1

Installed Plugins

No response

Java Version

bundled

OS Version

Linux f7ea544ab24a 5.15.49-linuxkit #1 SMP Tue Sep 13 07:51:46 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Problem Description

Starting in 8.7.0, the lucene query constructed for multi_match queries of type phrase_prefix and multiple explicitly specified fields is wrong, only searching one filed multiple times instead of all specified fields once, resulting in incomplete search results. See next section for detailed steps and description of where it goes wrong.

Steps to Reproduce

1. Create index

curl -X PUT "localhost:9200/people?pretty" -H 'Content-Type: application/json' -d'
{
  "settings": {
    "analysis": {
      "analyzer": {
        "default": {
          "type": "custom",
          "filter": [
            "lowercase",
            "asciifolding"
          ],
          "tokenizer": "whitespace"
        }
      }
    }
  },
  "mappings": {
    "dynamic_templates": [
      {
        "strings": {
          "match_mapping_type": "string",
          "mapping": {
            "type": "text",
            "fields": {
              "raw": {
                "type": "keyword"
              }
            },
            "index_prefixes": {
              "min_chars": 1,
              "max_chars": 19
            }
          }
        }
      }
    ]
  }
}
'

2. Add person to index

curl -X PUT "localhost:9200/people/_doc/1?pretty" -H 'Content-Type: application/json' -d'
{
  "first-name": "Harry",
  "last-name": "Potter",
  "email": "harry.potter@hogwarts.com",
  "identification": "01"
}
'

3. Search in all fields using multi_match and phrase_prefix, not find anything 🤔

curl -X GET "localhost:9200/people/_search?pretty" -H 'Content-Type: application/json' -d'
{
  "query": {
    "multi_match": {
      "query": "Har",
      "type": "phrase_prefix",
      "fields": [
        "first-name",
        "last-name",
        "email",
        "identification"
      ]
    }
  }
}
'

4. Show lucene query

curl -X GET "localhost:9200/people/_validate/query?rewrite=true" -H 'Content-Type: application/json' -d'
{
  "query": {
    "multi_match": {
      "query": "Har",
      "type": "phrase_prefix",
      "fields": [
        "first-name",
        "last-name",
        "email",
        "identification"
      ]
    }
  }
}
'

The resulting query searches the identification field four times. This is not right.

{
  "_shards": {
    "total": 1,
    "successful": 1,
    "failed": 0
  },
  "valid": true,
  "explanations": [
    {
      "index": "people",
      "valid": true,
      "explanation": "(identification._index_prefix:har | identification._index_prefix:har | identification._index_prefix:har | identification._index_prefix:har)"
    }
  ]
}

In ES 8.6.2 and older, the query searches all fields once.

{
  "_shards": {
    "total": 1,
    "successful": 1,
    "failed": 0
  },
  "valid": true,
  "explanations": [
    {
      "index": "people",
      "valid": true,
      "explanation": "(identification._index_prefix:har | first-name._index_prefix:har | last-name._index_prefix:har | email._index_prefix:har)"
    }
  ]
}

Logs (if relevant)

No response

[elasticsearchmachine]

Pinging @elastic/es-search (Team:Search)

[javanna]

Thanks a lot for reporting this! It's a subtle bug caused by a change in Lucene 9.5.0 (apache/lucene#11941). I will open a PR to fix this upstream.

[javanna]

I opened apache/lucene#12260 to address this in Lucene.