We suspected the same at first, and so modified the flow to do the copy of the new version "before" searching is activated on it. So, there are no searches on the new version dir at the time of copy - yet we still see slowness during the disk copy.
Thanks, - Gagan -----Original Message----- From: Anshum <ansh...@gmail.com> To: java-user@lucene.apache.org Sent: Mon, Aug 23, 2010 1:17 pm Subject: Re: slow search threads during a disk copy Seems like a case of I/O issues. You may be reading content off the index while performing searches while the I/O for copy is also happening. -- Anshum Gupta http://ai-cafe.blogspot.com On Mon, Aug 23, 2010 at 1:12 PM, <gag...@graffiti.net> wrote: > > Hi all, > > > We're observing search threads slowing down during directory copies > performed during updates to the index. The thread dump shows search threads > blocked on a FSDirectory$FSIndexInput$Descriptor instance: > > > > "Worker Thread - 12" daemon prio=10 tid=0x082b2400 nid=0x4654 waiting for > monitor entry [0x988ed000..0x988edf30] > java.lang.Thread.State: BLOCKED (on object monitor) > at > org.apache.lucene.store.FSDirectory$FSIndexInput.readInternal(FSDirectory.java:542) > - waiting to lock <0xc79c9900> (a > org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor) > at > org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:152) > at > org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:38) > at org.apache.lucene.store.IndexInput.readVInt(IndexInput.java:76) > at > org.apache.lucene.index.SegmentTermDocs.read(SegmentTermDocs.java:133) > at > org.apache.lucene.index.MultiSegmentReader$MultiTermDocs.read(MultiSegmentReader.java:573) > at org.apache.lucene.search.TermScorer.next(TermScorer.java:106) > at > org.apache.lucene.search.ConjunctionScorer.init(ConjunctionScorer.java:80) > at > org.apache.lucene.search.ConjunctionScorer.next(ConjunctionScorer.java:48) > at > org.apache.lucene.search.BooleanScorer2.score(BooleanScorer2.java:319) > at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:146) > at org.apache.lucene.search.Searcher.search(Searcher.java:118) > > ... > > > The directory copy we do is a 'cp -pr <src-dir>/* <dest-dir>' command prior > to applying changes (addDocument calls) on the current "available" segment. > This takes >7 mins to copy a directory of size 1.4G. During this time > window, the searches are slow and the above thread stacks are observed. > > > Could there be any system level limits we're hitting? > > > Our test environment is: > lucene-2.3.2 > 4x2.6 GHz, 16G memory > Red Hat 3.4.6-9 > > > Thanks and regards, > > > - Gagan >
