IW.getReader() returns inconsistent reader on RT Branch [LUCENE-3028]

[asfimport]

I extended the testcase TestRollingUpdates#testUpdateSameDoc to pull a NRT reader after each update and asserted that is always sees only one document. Yet, this fails with current branch since there is a problem in how we flush in the getReader() case. What happens here is that we flush all threads and then release the lock (letting other flushes which came in after we entered the flushAllThread context, continue) so that we could concurrently get a new segment that transports global deletes without the corresponding add. They sneak in while we continue to open the NRT reader which in turn sees inconsistent results.

I will upload a patch soon

---

Migrated from LUCENE-3028 by Simon Willnauer (@s1monw), resolved May 03 2011
Attachments: LUCENE-3028.patch (versions: 2), realtime-1.txt
Linked issues:

• Per thread DocumentsWriters that write their own private segments [LUCENE-2324] #3400

[asfimport]

Simon Willnauer (@s1monw) (migrated from JIRA)

here is a first patch

[asfimport]

Simon Willnauer (@s1monw) (migrated from JIRA)

next iteration, edited some asserts in DW

[asfimport]

Simon Willnauer (@s1monw) (migrated from JIRA)

I will commit this latest patch to the branch we can still iterates but since we have jenkins running builds I want to let that sink a bit too....

simon

[asfimport]

selckin (migrated from JIRA)

Seems to fail once every 6-8 runs quite consistently (at least i think this is the issue)

brachnes/realtime_search r1092329

{{
[junit] Testsuite: org.apache.lucene.index.TestRollingUpdates
[junit] Testcase: testUpdateSameDoc(org.apache.lucene.index.TestRollingUpdates): Caused an ERROR
[junit] MockDirectoryWrapper: cannot close: there are still open files: {_ho.fdt=1, _ho.prx=1, _ho.fdx=1, _ho.nrm=1, _j0.fdt=1, _ho.tis=1, _j0.fdx=1, _j0.tis=1, _j0.prx=1, _ho.frq=1, _ho.tvx=1, _ho.tvd=1, _j0.nrm=1, _ho.tvf=1, _j0.frq=1, _j0.tvf=1, _j0.tvd=1, _j0.tvx=1}
[junit] java.lang.RuntimeException: MockDirectoryWrapper: cannot close: there are still open files: {_ho.fdt=1, _ho.prx=1, _ho.fdx=1, _ho.nrm=1, _j0.fdt=1, _ho.tis=1, _j0.fdx=1, _j0.tis=1, _j0.prx=1, _ho.frq=1, _ho.tvx=1, _ho.tvd=1, _j0.nrm=1, _ho.tvf=1, _j0.frq=1, _j0.tvf=1
, _j0.tvd=1, _j0.tvx=1}
[junit] at org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:414)
[junit] at org.apache.lucene.index.TestRollingUpdates.testUpdateSameDoc(TestRollingUpdates.java:104)
[junit] at org.apache.lucene.util.LuceneTestCase$LuceneTestCaseRunner.runChild(LuceneTestCase.java:1226)
[junit] at org.apache.lucene.util.LuceneTestCase$LuceneTestCaseRunner.runChild(LuceneTestCase.java:1154)
[junit] Caused by: java.lang.RuntimeException: unclosed IndexInput
[junit] at org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:369)
[junit] at org.apache.lucene.store.Directory.openInput(Directory.java:122)
[junit] at org.apache.lucene.index.TermVectorsReader.<init>(TermVectorsReader.java:86)
[junit] at org.apache.lucene.index.SegmentReader$CoreReaders.openDocStores(SegmentReader.java:236)
[junit] at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:495)
[junit] at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:629)
[junit] at org.apache.lucene.index.IndexWriter$ReaderPool.getReadOnlyClone(IndexWriter.java:587)
[junit] at org.apache.lucene.index.DirectoryReader.<init>(DirectoryReader.java:172)
[junit] at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:377)
[junit] at org.apache.lucene.index.DirectoryReader.doReopenFromWriter(DirectoryReader.java:419)
[junit] at org.apache.lucene.index.DirectoryReader.doReopen(DirectoryReader.java:432)
[junit] at org.apache.lucene.index.DirectoryReader.reopen(DirectoryReader.java:392)
[junit] at org.apache.lucene.index.TestRollingUpdates$IndexingThread.run(TestRollingUpdates.java:129)
[junit]
[junit]
[junit] Testcase: testUpdateSameDoc(org.apache.lucene.index.TestRollingUpdates): FAILED
[junit] Some threads threw uncaught exceptions!
[junit] junit.framework.AssertionFailedError: Some threads threw uncaught exceptions!
[junit] at org.apache.lucene.util.LuceneTestCase.tearDown(LuceneTestCase.java:521)
[junit] at org.apache.lucene.util.LuceneTestCase$LuceneTestCaseRunner.runChild(LuceneTestCase.java:1226)
[junit] at org.apache.lucene.util.LuceneTestCase$LuceneTestCaseRunner.runChild(LuceneTestCase.java:1154)
[junit]
[junit]
[junit] Tests run: 2, Failures: 1, Errors: 1, Time elapsed: 6.649 sec
[junit]
[junit] ------------- Standard Error -----------------
[junit] NOTE: reproduce with: ant test -Dtestcase=TestRollingUpdates -Dtestmethod=testUpdateSameDoc -Dtests.seed=-4094951767438954769:-1203905293622856057
[junit] NOTE: reproduce with: ant test -Dtestcase=TestRollingUpdates -Dtestmethod=testUpdateSameDoc -Dtests.seed=-4094951767438954769:-1203905293622856057
[junit] The following exceptions were thrown by threads:
[junit] *** Thread: Thread-103 ***
[junit] java.lang.AssertionError: expected: org.apache.lucene.index.DocumentsWriterDeleteQueue@18635827but was: org.apache.lucene.index.DocumentsWriterDeleteQueue@223074f3 false
[junit] at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:359)
[junit] at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:346)
[junit] at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1367)
[junit] at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1339)
[junit] at org.apache.lucene.index.TestRollingUpdates$IndexingThread.run(TestRollingUpdates.java:125)
[junit] *** Thread: Thread-106 ***
[junit] java.lang.AssertionError: expected: org.apache.lucene.index.DocumentsWriterDeleteQueue@18635827but was: org.apache.lucene.index.DocumentsWriterDeleteQueue@223074f3 false
[junit] at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:359)
[junit] at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:346)
[junit] at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1367)
[junit] at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1339)
[junit] at org.apache.lucene.index.TestRollingUpdates$IndexingThread.run(TestRollingUpdates.java:125)
[junit] NOTE: test params are: codec=PreFlex, locale=th, timezone=America/St_Kitts
[junit] NOTE: all tests run in this JVM:
[junit] [TestSearch, TestCharTermAttributeImpl, TestCheckIndex, TestConsistentFieldNumbers, TestCrash, TestDeletionPolicy, TestDocumentWriter, TestIndexReaderCloneNorms, TestLongPostings, TestPayloads, TestPerFieldCodecSupport, TestRollingUpdates]
[junit] NOTE: Linux 2.6.37-gentoo amd64/Sun Microsystems Inc. 1.6.0_24 (64-bit)/cpus=8,threads=1,free=45768760,total=246743040
[junit] ------------- ---------------- ---------------
[junit] TEST org.apache.lucene.index.TestRollingUpdates FAILED
}}

[asfimport]

Simon Willnauer (@s1monw) (migrated from JIRA)

hmm I can't even after 1k runs :(

[asfimport]

Simon Willnauer (@s1monw) (migrated from JIRA)

I just committed a fix for this - seems like the assert which resets the current flushing queue was at the wrong position.

[asfimport]

selckin (migrated from JIRA)

hasn't failed since above fix

[asfimport]

Simon Willnauer (@s1monw) (migrated from JIRA)

hasn't failed since above fix

thanks for reporting back, the failure you reported was due to a reset call at the wrong position. I was allowing blocked flushed to continue before I reset the the var that ensures that the blocked flushes continue before a full flush finished.

[asfimport]

Simon Willnauer (@s1monw) (migrated from JIRA)

fixed in RT