Discover is not highlighting text in arrays of objects

[fdartayre]

Kibana version: 7.5.0

Describe the bug:
Kibana is not highlighting matches when the text field is inside an array of objets.

Steps to reproduce:

1. Create a new index with 2 documents

PUT test_index/_doc/1
{
  "field_a": { "field_b": "random string" }
}

PUT test_index/_doc/2
{
  "field_a": [
    { "field_b": "random string" }
  ]
}

2. Create the index pattern and search "string" in Discover

The "string" text should also be highlighted in the second document.

Expected behavior:
Here's the response from Elasticsearch to the _msearch request sent by Kibana:

{
  "took": 2,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 2,
    "max_score": 0,
    "hits": [
      {
        "_index": "test_index",
        "_type": "_doc",
        "_id": "1",
        "_version": 1,
        "_score": 0,
        "_source": {
          "field_a": {
            "field_b": "random string"
          }
        },
        "highlight": {
          "field_a.field_b": [
            "random @kibana-highlighted-field@string@/kibana-highlighted-field@"
          ]
        }
      },
      {
        "_index": "test_index",
        "_type": "_doc",
        "_id": "2",
        "_version": 1,
        "_score": 0,
        "_source": {
          "field_a": [
            {
              "field_b": "random string"
            }
          ]
        },
        "highlight": {
          "field_a.field_b": [
            "random @kibana-highlighted-field@string@/kibana-highlighted-field@"
          ]
        }
      }
    ]
  }
}

Elasticsearch is correctly highlighting "string" and Kibana should be showing it in Discover.

[elasticmachine]

Pinging @elastic/kibana-app (Team:KibanaApp)

[nickofthyme]

Hey @fdartayre,

I started to look into this issue and found that the formatter requires a field on the indexPattern in order to highlight the string. To do this it first flattens each of the hits _source and then tries to find the field on the indexPattern.

So in the case of _doc/1 we would get a flattened object like

{
  "field_a.field_b": "random string"
}

Which is fine because field_a.field_b is a filed type based on the implicit mappings. Thus the highlighting works.

In the case of doc/2 we would get a flattened object like

{
  "field_a": [
    {
      "field_b": "random string"
    }
  ]
}

But this time it tries to look for the field field_a which doesn't exist.

Even if it could find the field, the highlight payload doesn't say which index in the array the highlighted value is in. I think the only approach for this would be to use a nested type with the nested query with inner_hits, something like this...

GET test_index/_search
{
  "query": {
    "nested": {
      "path": "field_a",
      "query": {
        "bool": {
          "filter": [
            {
              "multi_match": {
                "type": "best_fields",
                "query": "string",
                "lenient": true
              }
            }
          ]
        }
      },
      "inner_hits": {
        "highlight": {
          "fields": {
            "*": {}
          }
        }
      }
    }
  }
}

Which would result in a response like below which would give you the information needed to highlight the nested fields, namely _nested.offset and _nested.field, for when there are multiple field_a in a single _doc.

{
  "took" : 2,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 2,
      "relation" : "eq"
    },
    "max_score" : 0.0,
    "hits" : [
      {
        "_index" : "test_index",
        "_id" : "1",
        "_score" : 0.0,
        "_source" : {
          "field_a" : {
            "field_b" : "random string"
          }
        },
        "inner_hits" : {
          "field_a" : {
            "hits" : {
              "total" : {
                "value" : 1,
                "relation" : "eq"
              },
              "max_score" : 0.0,
              "hits" : [
                {
                  "_index" : "test_index",
                  "_id" : "1",
                  "_nested" : {
                    "field" : "field_a",
                    "offset" : 0
                  },
                  "_score" : 0.0,
                  "_source" : {
                    "field_b" : "random string"
                  },
                  "highlight" : {
                    "field_a.field_b" : [
                      "random <em>string</em>"
                    ]
                  }
                }
              ]
            }
          }
        }
      },
      {
        "_index" : "test_index",
        "_id" : "2",
        "_score" : 0.0,
        "_source" : {
          "field_a" : [
            {
              "field_b" : "random string"
            }
          ]
        },
        "inner_hits" : {
          "field_a" : {
            "hits" : {
              "total" : {
                "value" : 1,
                "relation" : "eq"
              },
              "max_score" : 0.0,
              "hits" : [
                {
                  "_index" : "test_index",
                  "_id" : "2",
                  "_nested" : {
                    "field" : "field_a",
                    "offset" : 0
                  },
                  "_score" : 0.0,
                  "_source" : {
                    "field_b" : "random string"
                  },
                  "highlight" : {
                    "field_a.field_b" : [
                      "random <em>string</em>"
                    ]
                  }
                }
              ]
            }
          }
        }
      }
    ]
  }
}

All that said, it seems to me that there was never support for the nested field highlighting but I can certainly be wrong. I am checking with the team to be sure but wanted to ask if you know of an older version of kibana that supported this functionality?

[ibtehajn]

Hi,

I am seeing the same issue. The fact that the discover tab doesn't correctly recognize fields of sub-objects wrapped in arrays is clearly a bug itself (the highlighting issue is just one problem that arises as a result, but this is not the root cause nor the only issue). Other problems e.g. are that you cannot use the "filter for value" button on affected fields. See also this thread.

I think this bug needs to be looked into, but not in the context of highlighting search results. In general, the suggestion of using a nested mapping is not a solution.

[elasticmachine]

Pinging @elastic/kibana-app-services (Team:AppServices)

[elasticmachine]

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

[kertal]

related to #3735

[jughosta]

With fields API it works now correctly (when discover:searchFieldsFromSource is off):