Expose iterator over query terms in TermInSetQuery

[gsmiller]

Description

I'd like to propose we add an API to TermInSetQuery that exposes an iterator over the query terms. This is useful for extending TermInSetQuery. One concrete use-case for this is needing to change the way term intersection happens with the indexed terms dictionary to support bloom filters, as described in this email thread.

I don't think there's any harm in exposing this, but am interested in feedback of course! This abstraction decouples the current prefix-coding implementation details, so it seems clean.

[rmuir]

Please lets not go this path. It is a mutlitermquery, if you want to change how it works behind the scenes, you "plugin" with RewriteMethod

[gsmiller]

Thanks @rmuir. It would be ideal if we could do this through RewriteMethod, but I'm not sure how we can actually accomplish that. The problem is in the implementation of getTerms, as defined in TermInSetQuery. We can plug in through RewriteMethod#getTerms, but we still need access to an iterator of the query terms. I'll draft up a sandbox query that might help illustrate the issue and we can discuss further. If there's a better way to go about this, happy to explore it. Thanks again!

Update: I added a sandbox query to this PR just to demonstrate the use-case for extending TermInSetQuery and overriding getTerms. It shows how this proposed getQueryTerms method would be used. Happy to consider other ways we might support this sort of use-case. Thanks again.

[gsmiller]

This class is for demo purposes only. I'm not suggesting we merge it as part of this PR. I only want to demonstrate how a class might leverage getQueryTerms.

[rmuir]

I still think it doesn't make sense to me to expose this. As i said on the dev list, your problem is that you use a custom postings format and you want it to accelerate the intersection.

The cleanest way to do this, is to handoff the intersection to the postingsformat directly, rather than worry about seekCeil/seekExact and subclassing queries or exposing stuff. It should give a performance improvement using the default postings format as well (at least it did for other queries when mikemccand added it)

So, IMO we should try to fix this query to use Terms.intersect() [see #12176], then override Terms.intersect for the BloomPostingsFormat to make use of the bloom filters to speed up intersection.

[gsmiller]

Got it, thanks @rmuir. I hadn't seen your dev list reply yet. This all makes sense. I'll close this out and have a look at leveraging intersect. Seems like a better path forward. Thanks!