Well, there still sounds like plenty of scope for IO contention. Are source and dest directories on the same disk? What does iostat show? Could you use rsync rather than cp -rp, or something else that only copies changes?
-- Ian. On Mon, Aug 23, 2010 at 9:20 AM, <gag...@graffiti.net> wrote: > > We suspected the same at first, and so modified the flow to do the copy of > the new version "before" searching is activated on it. So, there are no > searches on the new version dir at the time of copy - yet we still see > slowness during the disk copy. > > > Thanks, > - Gagan > > > > > > -----Original Message----- > From: Anshum <ansh...@gmail.com> > To: java-user@lucene.apache.org > Sent: Mon, Aug 23, 2010 1:17 pm > Subject: Re: slow search threads during a disk copy > > > Seems like a case of I/O issues. You may be reading content off the index > while performing searches while the I/O for copy is also happening. > > -- > Anshum Gupta > http://ai-cafe.blogspot.com > > > On Mon, Aug 23, 2010 at 1:12 PM, <gag...@graffiti.net> wrote: > >> >> Hi all, >> >> >> We're observing search threads slowing down during directory copies >> performed during updates to the index. The thread dump shows search threads >> blocked on a FSDirectory$FSIndexInput$Descriptor instance: >> >> >> >> "Worker Thread - 12" daemon prio=10 tid=0x082b2400 nid=0x4654 waiting for >> monitor entry [0x988ed000..0x988edf30] >> java.lang.Thread.State: BLOCKED (on object monitor) >> at >> org.apache.lucene.store.FSDirectory$FSIndexInput.readInternal(FSDirectory.java:542) >> - waiting to lock <0xc79c9900> (a >> org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor) >> at >> org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:152) >> at >> org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:38) >> at org.apache.lucene.store.IndexInput.readVInt(IndexInput.java:76) >> at >> org.apache.lucene.index.SegmentTermDocs.read(SegmentTermDocs.java:133) >> at >> org.apache.lucene.index.MultiSegmentReader$MultiTermDocs.read(MultiSegmentReader.java:573) >> at org.apache.lucene.search.TermScorer.next(TermScorer.java:106) >> at >> org.apache.lucene.search.ConjunctionScorer.init(ConjunctionScorer.java:80) >> at >> org.apache.lucene.search.ConjunctionScorer.next(ConjunctionScorer.java:48) >> at >> org.apache.lucene.search.BooleanScorer2.score(BooleanScorer2.java:319) >> at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:146) >> at org.apache.lucene.search.Searcher.search(Searcher.java:118) >> >> ... >> >> >> The directory copy we do is a 'cp -pr <src-dir>/* <dest-dir>' command prior >> to applying changes (addDocument calls) on the current "available" segment. >> This takes >7 mins to copy a directory of size 1.4G. During this time >> window, the searches are slow and the above thread stacks are observed. >> >> >> Could there be any system level limits we're hitting? >> >> >> Our test environment is: >> lucene-2.3.2 >> 4x2.6 GHz, 16G memory >> Red Hat 3.4.6-9 >> >> >> Thanks and regards, >> >> >> - Gagan >> > > > > --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
