Han-Wen NIenhuys created LUCENE-7499:
----------------------------------------
Summary: IndexWriter race condition
Key: LUCENE-7499
URL: https://issues.apache.org/jira/browse/LUCENE-7499
Project: Lucene - Core
Issue Type: Bug
Components: core/index
Affects Versions: 6.2, 5.5.2
Reporter: Han-Wen NIenhuys
Priority: Minor
separated out from https://issues.apache.org/jira/browse/LUCENE-2585
I've seen race condition diagnostics for Lucene code that could be related to
this. Line numbers are for Lucene 5.5.2, but the problem seems present in
master too.
AFAICT: StandardDirectoryReader has an IndexWriter, and it checks things are
up-to-date by checking if there are pending in-memory updates by looking at
IndexWriter.SegmentInfos.version. That read is not synchronized relative to its
write from IndexWriter.updateDocument
I'm not enough of a lucene expert to suggest how to fix this in detail. If I
would guess at how Lucene roughly works, I would suggest the following
* when applying pending in-memory changes, this takes a lock, and under lock
there is a "changed = true" assignment.
* when checking for changes, take lock, return changed.
* when writing pending in-memory changes, take lock, write out file, set
"changed=false", release lock.
The problem can be reproduced by running some of the Gerrit acceptance tests
under Threadsanitizer. I don't know of open-source tooling to set that up,
unfortunately.
WARNING: ThreadSanitizer: data race (pid=21636)
Write of size 8 at 0x7f24ad9d3210 by thread T36 (mutexes: write
M431922409782456832):
#0 org.apache.lucene.index.SegmentInfos.changed()V (SegmentInfos.java:944)
#1 org.apache.lucene.index.IndexWriter.newSegmentName()Ljava/lang/String;
(IndexWriter.java:1652)
#2
org.apache.lucene.index.DocumentsWriter.ensureInitialized(Lorg/apache/lucene/index/DocumentsWriterPerThreadPool$ThreadState;)V
(DocumentsWriter.java:391)
#3
org.apache.lucene.index.DocumentsWriter.updateDocument(Ljava/lang/Iterable;Lorg/apache/lucene/analysis/Analyzer;Lorg/apache/lucene/index/Term;)Z
(DocumentsWriter.java:445)
#4
org.apache.lucene.index.IndexWriter.updateDocument(Lorg/apache/lucene/index/Term;Ljava/lang/Iterable;)V
(IndexWriter.java:1477)
#5
com.google.gerrit.lucene.AutoCommitWriter.updateDocument(Lorg/apache/lucene/index/Term;Ljava/lang/Iterable;)V
(AutoCommitWriter.java:100)
#6
org.apache.lucene.index.TrackingIndexWriter.updateDocument(Lorg/apache/lucene/index/Term;Ljava/lang/Iterable;)J
(TrackingIndexWriter.java:55)
#7 com.google.gerrit.lucene.AbstractLuceneIndex$4.call()Ljava/lang/Long;
(AbstractLuceneIndex.java:250)
#8 com.google.gerrit.lucene.AbstractLuceneIndex$4.call()Ljava/lang/Object;
(AbstractLuceneIndex.java:247)
#9
com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly()V
(TrustedListenableFutureTask.java:108)
#10 com.google.common.util.concurrent.InterruptibleTask.run()V
(InterruptibleTask.java:41)
#11 com.google.common.util.concurrent.TrustedListenableFutureTask.run()V
(TrustedListenableFutureTask.java:77)
#12
java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V
(ThreadPoolExecutor.java:1142)
#13 java.util.concurrent.ThreadPoolExecutor$Worker.run()V
(ThreadPoolExecutor.java:617)
#14 java.lang.Thread.run()V (Thread.java:745)
#15 (Generated Stub)
Previous read of size 8 at 0x7f24ad9d3210 by thread T29 (mutexes: write
M1060737507754061632):
#0
org.apache.lucene.index.IndexWriter.nrtIsCurrent(Lorg/apache/lucene/index/SegmentInfos;)Z
(IndexWriter.java:4592)
#1
org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(Lorg/apache/lucene/index/IndexCommit;)Lorg/apache/lucene/index/DirectoryReader;
(StandardDirectoryReader.java:282)
#2
org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(Lorg/apache/lucene/index/IndexCommit;)Lorg/apache/lucene/index/DirectoryReader;
(StandardDirectoryReader.java:261)
#3
org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged()Lorg/apache/lucene/index/DirectoryReader;
(StandardDirectoryReader.java:251)
#4
org.apache.lucene.index.DirectoryReader.openIfChanged(Lorg/apache/lucene/index/DirectoryReader;)Lorg/apache/lucene/index/DirectoryReader;
(DirectoryReader.java:137)
#5
com.google.gerrit.lucene.WrappableSearcherManager.refreshIfNeeded(Lorg/apache/lucene/search/IndexSearcher;)Lorg/apache/lucene/search/IndexSearcher;
(WrappableSearcherManager.java:148)
#6
com.google.gerrit.lucene.WrappableSearcherManager.refreshIfNeeded(Ljava/lang/Object;)Ljava/lang/Object;
(WrappableSearcherManager.java:68)
#7 org.apache.lucene.search.ReferenceManager.doMaybeRefresh()V
(ReferenceManager.java:176)
#8 org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking()V
(ReferenceManager.java:253)
#9 org.apache.lucene.search.ControlledRealTimeReopenThread.run()V
(ControlledRealTimeReopenThread.java:245)
#10 (Generated Stub)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
