Seems like a case of I/O issues. You may be reading content off the index while performing searches while the I/O for copy is also happening.
-- Anshum Gupta http://ai-cafe.blogspot.com On Mon, Aug 23, 2010 at 1:12 PM, <gag...@graffiti.net> wrote: > > Hi all, > > > We're observing search threads slowing down during directory copies > performed during updates to the index. The thread dump shows search threads > blocked on a FSDirectory$FSIndexInput$Descriptor instance: > > > > "Worker Thread - 12" daemon prio=10 tid=0x082b2400 nid=0x4654 waiting for > monitor entry [0x988ed000..0x988edf30] > java.lang.Thread.State: BLOCKED (on object monitor) > at > org.apache.lucene.store.FSDirectory$FSIndexInput.readInternal(FSDirectory.java:542) > - waiting to lock <0xc79c9900> (a > org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor) > at > org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:152) > at > org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:38) > at org.apache.lucene.store.IndexInput.readVInt(IndexInput.java:76) > at > org.apache.lucene.index.SegmentTermDocs.read(SegmentTermDocs.java:133) > at > org.apache.lucene.index.MultiSegmentReader$MultiTermDocs.read(MultiSegmentReader.java:573) > at org.apache.lucene.search.TermScorer.next(TermScorer.java:106) > at > org.apache.lucene.search.ConjunctionScorer.init(ConjunctionScorer.java:80) > at > org.apache.lucene.search.ConjunctionScorer.next(ConjunctionScorer.java:48) > at > org.apache.lucene.search.BooleanScorer2.score(BooleanScorer2.java:319) > at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:146) > at org.apache.lucene.search.Searcher.search(Searcher.java:118) > > ... > > > The directory copy we do is a 'cp -pr <src-dir>/* <dest-dir>' command prior > to applying changes (addDocument calls) on the current "available" segment. > This takes >7 mins to copy a directory of size 1.4G. During this time > window, the searches are slow and the above thread stacks are observed. > > > Could there be any system level limits we're hitting? > > > Our test environment is: > lucene-2.3.2 > 4x2.6 GHz, 16G memory > Red Hat 3.4.6-9 > > > Thanks and regards, > > > - Gagan >
