API: Add response filtering with filter_path parameter

[tlrx]

This change adds a new "_path" parameter that can be used to filter and reduce the responses returned by the REST API of elasticsearch.

For example, returning only the shards that failed to be optimized:

curl -XPOST 'localhost:9200/beer/_optimize?_path=_shards.failed'
{"_shards":{"failed":0}}%

It supports multiple filters (separated by a comma):

curl -XGET 'localhost:9200/_mapping?pretty&_path=*.mappings.*.properties.name,*.mappings.*.properties.title'

It also supports the YAML response format. Here it returns only the _id field of a newly indexed document:

curl -XPOST 'localhost:9200/library/book?_path=_id' -d '---hello:\n  world: 1\n'

---
_id: "AU0j64-b-stVfkvus5-A"

It also supports wildcards. Here it returns only the host name of every nodes in the cluster:

curl -XGET 'http://localhost:9200/_nodes/stats?_path=nodes.*.host*'
{"nodes":{"lvJHed8uQQu4brS-SXKsNA":{"host":"portable"}}}

And "**" can be used to include sub fields without knowing the exact path. Here it returns only the Lucene version of every segment:

curl 'http://localhost:9200/_segments?pretty&_path=indices.**.version'
{
  "indices" : {
    "beer" : {
      "shards" : {
        "0" : [ {
          "segments" : {
            "_0" : {
              "version" : "5.2.0"
            },
            "_1" : {
              "version" : "5.2.0"
            }
          }
        } ]
      }
    }
  }
}

Note that elasticsearch sometimes returns directly the raw value of a field, like the "_source" field. If you want to filter the response that include _source fields like in Search or Get responses , you should consider using the already existing "fields" parameter:

 curl -XGET 'localhost:9200/beer/tasty/3672?pretty&fields=category'
 {
   "_index" : "beer",
   "_type" : "tasty",
   "_id" : "3672",
   "_version" : 1,
   "found" : true,
   "fields" : {
     "category" : [ "German Lager" ]
   }
 }

The "_path" parameter can be used to further reduce the result:

curl -XGET 'localhost:9200/beer/tasty/3672?pretty&fields=category&_path=fields.category'
{
  "fields" : {
    "category" : [ "German Lager" ]
  }
}

Closes #7401

[tlrx]

Note:

• if the provided filter specified in _path does not match something, an empty response is returned
• if the API returns an error, the error is not filtered even if the _path is specified

[clintongormley]

if the provided filter specified in _path does not match something, an empty response is returned

We should return an empty JSON object rather than an empty string, otherwise the client needs to check whether JSON has been returned.

[tlrx]

@clintongormley thanks, I updated the code to reflect your last comment.

[tlrx]

@spinscale Can you please have a look? Thanks

[spinscale]

use CollectionUtils.isEmpty()?

[tlrx]

Ok

[spinscale]

A couple of things

First, I really would like to see some simple benchmarking, so we have some ballpark numbers, how this affects performance. Especially because write operations that create sub-hierarchies also create new objects.

Minor thing: FilteringJsonGenerator.parse() - why not use an array instead of XContentFilter and next (like your own custom linked list).

One other thing, I know that jackson has some filtering capabiltiies built-in, I have no ideas if we can make use of them though, see here, here and here

[tlrx]

One other thing, I know that jackson has some filtering capabiltiies built-in, I have no ideas if we can make use of them though, see here, here and here

That's the first thing we checked before starting working on this pull request. The current filtering capabilities of Jackson (version <2.5.1) can be used only with jackson-databinding lib and apply only to POJOs but not when you use jackson to generate JSON in streaming mode.

Jackson 2.6 will integrate an interesting filtering feature that will work with JSON streaming (see release note here and test class here). It currently uses JSONPointer to filter properties but I think we will be able to implement our own TokenFilter later, once version 2.6 is released.

[tlrx]

First, I really would like to see some simple benchmarking, so we have some ballpark numbers, how this affects performance. Especially because write operations that create sub-hierarchies also create new objects.

I added the FilteringJsonGeneratorBenchmark and put some numbers here (not very readable - I apologize for that).

There are no real surprise in the benchmark:

• when writing a small number of documents (less than 100) with a small number of fields each(1-10), the filtering adds an overhead that makes it less efficient than what we have today
• when writing a larger amount of data (ie large number of small docs, small number of large docs) there's a point where it becomes more efficient to check if fields must be included and skip them rather than writing more data. This point varies depending on the document size and the selectivity of the filter, but in this test it is in between 1-10%.

Minor thing: FilteringJsonGenerator.parse() - why not use an array instead of XContentFilter and next (like your own custom linked list).

There are room for improvements like this one (I'm also thinking of not creating FilterContext for sub fields when a parent field matches the end of a filter), but it might add complexity to the code and I'm not sure if it will be really more efficient.

I'm currently trying some of these improvement to see if they are pertinent or not.

[tlrx]

Thinking of improvements again, we should be able to reuse the objects instead of creating new ones for every sub fields. I'll try to improve that.

[clintongormley]

when writing a small number of documents (less than 100) with a small number of fields each(1-10), the filtering adds an overhead that makes it less efficient than what we have today

I assume that the code path (and performance) remains the same as today if no _path is specified?

[tlrx]

@clintongormley yes. Branching is done here where we fall back to the normal XContent generator.

[spinscale]

... written

[spinscale]

added some absolutely minor comments...

First, this doesnt compile using mvn, due to the benchmark class and some locale, but not really an issue.

Few last things:

• the kind-of linkedlist data structure in the FilterContext, which could possibly be replaced by a simple array?
• This seems the only part, where we use ** as 'greedy' wildcard in Elasticsearch, not sure if it is a problem, just to be aware
• No documentation so far?

Maybe we should also note somewhere to switch to the jackson filter features with the 2.6 release, I'd hate to forget that...

LGTM apart from that

[rjernst]

I'm late to the party here, but I have a simple question. Why is the parameter _path and not path? Why use the underscore when we control all the url params anyways? And all the other examples I know of (eg fields, pretty, human) do not use underscore?

[clintongormley]

@rjernst only because it works well with the existing _source filtering. I kinda like the underscore because it is a "global" param (mind you, so is pretty).

[clintongormley]

This seems the only part, where we use ** as 'greedy' wildcard in Elasticsearch, not sure if it is a problem, just to be aware

Elsewhere we use * as a greedy wildcard, which I think is problematic. We took the ** syntax from ant, and I'd actually be tempted to reuse it elsewhere.

[kimchy]

actually, since this is a "rest" level parameter, and for those, we have pretty and human, I think we should just call it path, or maybe be more explicit and call it filter_path

[rjernst]

+1 on filter_path

[kimchy]

here is another suggestion, calling it filter_path_include, in the future we might want to support filter_path_exclude.

The reason I am raising it is that today, we support source level include/exclude when fetching source documents. With this infrastructure, if we eventually expand it to support include_s_/exclude_s_, we can also support zero copy (as in, not create a map of maps/lists representation of source) when someone asks for source includes/includes (we can just create a bytes based filtering generator, and copy structure from the parser right into it).

I helped a fellow on IRC today where the loading and parsing of source include was the perf bottleneck (from 1ms it got up to 170ms for ~1000 docs). I think this will help a lot there eventually.

[tlrx]

@spinscale I rebased and updated the code according to your last comments. Can you have a look and do some manual testing please? Thanks a lot :)

Note: latest benchmark numbers are here.

[spinscale]

we may also not want to have manual testing but also add that parameter (plus tests) to our REST tests

[tlrx]

@spinscale I just added the REST tests. The parameter is still named _path until we reach an agreement on the final name.

[spinscale]

I think the _path needs to be added to all the methods supporting it, to make sure the clients are aware

[clintongormley]

No, like pretty and _source, we can just support this parameter everywhere, so don't add it to the rest spec.

[spinscale]

left one comment, apart from that it feels like we are close to getting this in, when we get the naming stuff resolved... /cc @clintongormley

[clintongormley]

The _source parameter is essentially a synonym for _source_include, so I'd go with filter_path for now. We can always add filter_path_include|exclude later on.

[tlrx]

@clintongormley sounds good to me, thanks

[tlrx]

@spinscale Updated with your last comment :)

[clintongormley]

@spinscale @kimchy does this need another review, or are we good to go?

[spinscale]

LGTM on my side

[clintongormley]

w00t

[rashidkpc]

WIN Thanks so much everyone for making this happen. This is huge for Kibana!

[dkroehan]

Is this already supported via the Java API?

[tlrx]

@dkroehan The filter_path parameter is used as the REST level (like the pretty parameter) to render JSON responses and therefore it is not part of the Java API.

If you need to render XContent object instances using filtering in Java, you can use the XContent.builder(XContent xContent, String[] filters) method

[pickworth]

are excludes supported yet, or just includes?

[tlrx]

@nmors Using the REST API and the filter_path parameter, only includes are supported. Using XContent filtering in Java, excludes are supported but you can't mix them both.

[pickworth]

Thanks, I'm actually using the javascript client. I'm using the filterPath parameter like so (I guess it's a wrapper for filter_path), all is working fine! It would be good to be able to use excludes though, can be very helpful for me as it is web traffic that I would like to keep small. I've managed to cut the response size in half by using the following:

  index: my_index,
  type: my_type,
  body: q,
  filterPath: [
    "aggregations.my_agg.buckets.**.value",
    "aggregations.my_agg.buckets.**.key"
  ]