Using IndexSearcher in a multithreaded context causes contention [LUCENE-1309]

[asfimport]

The attached code allows to search with a single IndexSearcher across threads or with a single instance per Thread, additionally it allows to deals with String.intern in Field from #2385.

Really interesting is the behavior of the current implementation (Field String.intern) The performance improves dramatically.

Is this by design? Because it is usually recommended to reuse the IndexSearcher. How about a version to supports mult-threading much better basically delivering a read-only view?

The following results have been measured. The [main] part is a single threaded run. The following thread results

Field String.intern, Single Searcher
[main] Search took: 3453ms
[Thread-2] Search took: 17812ms
[Thread-3] Search took: 18313ms
[Thread-1] Search took: 18234ms
[Thread-0] Search took: 18562ms

Field WeakHashMap, Single Searcher
[main] Search took: 3156ms
[Thread-3] Search took: 14953ms
[Thread-1] Search took: 15593ms
[Thread-0] Search took: 15656ms
[Thread-2] Search took: 16188ms

Field ConcurrentHashMap, Single Searcher
[main] Search took: 2844ms
[Thread-1] Search took: 14812ms
[Thread-0] Search took: 14890ms
[Thread-2] Search took: 15172ms
[Thread-3] Search took: 14656ms

Field intern, Searcher per Thread
[main] Search took: 3687ms
[Thread-2] Search took: 10766ms
[Thread-1] Search took: 10734ms
[Thread-0] Search took: 11047ms
[Thread-3] Search took: 10938ms

Field Weak, Searcher per Thread
[main] Search took: 3187ms
[Thread-3] Search took: 11625ms
[Thread-2] Search took: 12484ms
[Thread-1] Search took: 12609ms
[Thread-0] Search took: 12391ms

Field Concurrent, Searcher per Thread
[main] Search took: 2938ms
[Thread-3] Search took: 10203ms
[Thread-2] Search took: 10360ms
[Thread-1] Search took: 10406ms
[Thread-0] Search took: 10719ms

---

Migrated from LUCENE-1309 by Rene Schwietzke, resolved Nov 17 2008
Environment:

* WinXP, Dual Core Centrino
* Opensuse Linux, Dual Core AMD

java version "1.6.0_06"
Java(TM) SE Runtime Environment (build 1.6.0_06-b02)
Java HotSpot(TM) Client VM (build 10.0-b22, mixed mode, sharing)

Attachments: yad.zip
Linked issues:

• Remove String.intern() from Field.java to increase performance and lower contention [LUCENE-1308] #2385
• Identify bottleneck associated with not pooling searchers [LUCENE-1265] #2342

[asfimport]

Yonik Seeley (@yonik) (migrated from JIRA)

How many threads were used to do this test?

Because it is usually recommended to reuse the IndexSearcher

Yes (actually the IndexReader is the important part to cache and re-use).
Because of memory usage associated with an IndexReader, it would be much better to improve or fix multi-threaded behavior rather than use multiple IndexReaders.

[asfimport]

Rene Schwietzke (migrated from JIRA)

The above results were created running four threads.

[asfimport]

Michael McCandless (@mikemccand) (migrated from JIRA)

Dup of #2406 (readOnly IndexReader) and #1828 (NIOFSDirectory), at least on non-Windows platforms.

Based on this user report it seems like sharing an IndexReader across multiple threads in fact gives better performance than thread-private IndexReader instances:

http://www.nabble.com/IndexSearcher-and-multi-threaded-performance-td20449375.html