Highlight Field Order

[cexmmaqsood]

Elasticsearch version: 1.7

Plugins installed: Mapper Attachments

OS version: Sierra

Description of the problem including expected versus actual behaviour:
In https://www.elastic.co/guide/en/elasticsearch/reference/1.7/search-request-highlighting.html, it states that by setting order:score, my highlight results will be ordered in terms of relevancy (score). But how is that done considering the result is a highlight object that is a JSON object?

I've also tried it, and my fields do not get sorted by relevancy. I have the following two fields:

dateTime: 2017-02-02
Mapping Details:
{ type: 'string', analyzer: 'formatted' };

textBox: 2017 2017 2017 2017
Mapping Details:
{ type: 'string', analyzer: 'formatted' };

Custom Analyzers
'formatted': {
	'tokenizer': 'standard',
	'filter': ['length']
},
filter: {
	'length': {
	 'type': 'length',
	 'min': '2'
	}
}

When I perform the following search:

{
  "query": {
    "bool": {
      "should": [
        [
          {
            "bool": {
              "must": [
                {
                  "multi_match": {
                    "query": "2017",
                    "fields": [
                      "testTextAreaQA__formatted",
                      "lastUpdatedDate__formatted"
                    ],
                    "type": "best_fields",
                    "lenient": true,
                    "minimum_should_match": "100%"
                  }
                }
              ]
            }
          }
        ]
      ]
    }
  },
  "from": 0,
  "highlight": {
    "fields": [
      {
        "testTextAreaQA__formatted._highlight": {}
      },
      {
        "lastUpdatedDate__formatted._highlight": {}
      }
    ]
  }
}

I get the following highlighting return. Note that last updated date is before testTextArea, even though there's more matches in testTextArea.

"highlight": {
   "lastUpdatedDate__formatted._highlight": [
      "<em>2017</em>-09-12T21:03:47.897Z"
   ],
   "testTextAreaQA__formatted._highlight": [
      "<em>2017</em> <em>2017</em> <em>2017</em> <em>2017</em>"
   ]
}

Steps to reproduce:

1. Create above mappings
2. Search using above query

Provide logs (if relevant):

[colings86]

@nik9000 could you take a look at this?

[nik9000]

Highlighting doesn't have an option to sort the fields returned in any way.

The "sort": "score" option allows you to sort the fragments returned in each field. This only comes into play when you set number_of_fragments to something more than 1.

Sorry we don't have the thing you want! We certainly could make something like that but we aren't really doing a ton of work on highlighters and comparing the scores across different fields is a somewhat fraught thing.

[cexmmaqsood]

@nik9000 Thanks so much for the response! Do you think it's a good idea to make the documentation a bit more clear on what the sorting means and results in? Just to prevent further misunderstandings like this.

[nik9000]

@nik9000 Thanks so much for the response! Do you think it's a good idea to make the documentation a bit more clear on what the sorting means and results in? Just to prevent further misunderstandings like this.

Sure! Are you interested in fixing it or should I add this to my TODO list?

[cexmmaqsood]

@nik9000 I'd be glad to do it. A few more questions though, I don't see in the documentation anywhere if it states that sort:score is the only option, so I'm wondering if it is. As well, is the default fragment sorting by score? Because that is the behaviour I'm getting.

[nik9000]

I don't believe it is the default. At least it wasn't the default when I last used it which was a few years ago. I think the default is the order that the fragments appear in the document.

[CExAdamJ]

@nik9000 From reading this https://www.elastic.co/guide/en/elasticsearch/reference/1.7/search-request-highlighting.html it seems as though the property is "order" and not "sort". It seems to be consistent from 1.7 through to 6.2. Are these two different things or a mistake somewhere along the line?

[nik9000]

I suspect I've just made a mistake when remembering the name of the setting.

[CExAdamJ]

Ah ok, thanks. It happens :)

[allenwars]

@nik9000 Thanks so much for the response! Do you think it's a good idea to make the documentation a bit more clear on what the sorting means and results in? Just to prevent further misunderstandings like this.

Sure! Are you interested in fixing it or should I add this to my TODO list?

Hi @nik9000 I also want this functionality for my project, is it built yet?, if it is please let me know, if not then I am more than happy to take this issue to my bucket.

[nik9000]

Hi @nik9000 I also want this functionality for my project, is it built yet?, if it is please let me know, if not then I am more than happy to take this issue to my bucket.

No one's worked on it.