IndexWriter.deleteDocuments should have option to not apply to docs indexed in the current IW session [LUCENE-2679]

[asfimport]

In #3729 we are struggling with how to handle buffered deletes,
with the new per-thread RAM buffers (DWPT).

But, the only reason why we must maintain a map of del term -> current
docID (or sequence ID) is to correctly handle the interleaved adds &
deletes case.

However, I suspect that for many apps that interleaving never happens.
Ie, most apps delete only docs from before the last commit or NRT
reopen. For such apps, we don't need a Map... we just need a Set of
all del terms to apply to past segments but not to the currently
buffered docs.

And, importantly, with #3729, this would be a single Set, not
one per DWPT. It should be a a healthy RAM reduction on buffered
deletes, and should make the deletes call faster (add to one set instead of
N maps).

We of course must still support the interleaved case, and I think it
should be the default, but I think we should provide the option for
the common-case apps to take advantage of much less RAM usage.

---

Migrated from LUCENE-2679 by Michael McCandless (@mikemccand), updated May 09 2016

[asfimport]

Steven Rowe (@sarowe) (migrated from JIRA)

Bulk move 4.4 issues to 4.5 and 5.0

[asfimport]

Rodrigo Vega (migrated from JIRA)

I actually have "interleaved adds & deletes " and it fails kind of randomly. There is any work around i can use?

[asfimport]

Michael McCandless (@mikemccand) (migrated from JIRA)

It should not fail, it should work. Can you post a test case to the Lucene user's (java-user@lucene.apache.org) list?

[asfimport]

Uwe Schindler (@uschindler) (migrated from JIRA)

Move issue to Lucene 4.9.