Yes, all version directories are on the same disk. iostat output should be useful. Using rsync is complicated because of legacy reasons - there's a change impact to manage.
Intererstingly, the copy is quite fast (around 30s) when there are no searches in progress. Thanks, - Gagan -----Original Message----- From: Ian Lea <ian....@gmail.com> To: java-user@lucene.apache.org Sent: Mon, Aug 23, 2010 2:10 pm Subject: Re: slow search threads during a disk copy Well, there still sounds like plenty of scope for IO contention. Are source and dest directories on the same disk? What does iostat show? Could you use rsync rather than cp -rp, or something else that only copies changes? -- Ian. On Mon, Aug 23, 2010 at 9:20 AM, <gag...@graffiti.net> wrote: > > We suspected the same at first, and so modified the flow to do the copy of > the new version "before" searching is activated on it. So, there are no searches on the new version dir at the time of copy - yet we still see slowness during the disk copy. > > > Thanks, > - Gagan > > > > > > -----Original Message----- > From: Anshum <ansh...@gmail.com> > To: java-user@lucene.apache.org > Sent: Mon, Aug 23, 2010 1:17 pm > Subject: Re: slow search threads during a disk copy > > > Seems like a case of I/O issues. You may be reading content off the index > while performing searches while the I/O for copy is also happening. > > -- > Anshum Gupta > http://ai-cafe.blogspot.com > > > On Mon, Aug 23, 2010 at 1:12 PM, <gag...@graffiti.net> wrote: > >> >> Hi all, >> >> >> We're observing search threads slowing down during directory copies >> performed during updates to the index. The thread dump shows search threads >> blocked on a FSDirectory$FSIndexInput$Descriptor instance: >> >> >> >> "Worker Thread - 12" daemon prio=10 tid=0x082b2400 nid=0x4654 waiting for >> monitor entry [0x988ed000..0x988edf30] >> java.lang.Thread.State: BLOCKED (on object monitor) >> at >> org.apache.lucene.store.FSDirectory$FSIndexInput.readInternal(FSDirectory.java:542) >> - waiting to lock <0xc79c9900> (a >> org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor) >> at >> org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:152) >> at >> org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:38) >> at org.apache.lucene.store.IndexInput.readVInt(IndexInput.java:76) >> at >> org.apache.lucene.index.SegmentTermDocs.read(SegmentTermDocs.java:133) >> at >> org.apache.lucene.index.MultiSegmentReader$MultiTermDocs.read(MultiSegmentReader.java:573) >> at org.apache.lucene.search.TermScorer.next(TermScorer.java:106) >> at >> org.apache.lucene.search.ConjunctionScorer.init(ConjunctionScorer.java:80) >> at >> org.apache.lucene.search.ConjunctionScorer.next(ConjunctionScorer.java:48) >> at >> org.apache.lucene.search.BooleanScorer2.score(BooleanScorer2.java:319) >> at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:146) >> at org.apache.lucene.search.Searcher.search(Searcher.java:118) >> >> ... >> >> >> The directory copy we do is a 'cp -pr <src-dir>/* <dest-dir>' command prior >> to applying changes (addDocument calls) on the current "available" segment. >> This takes >7 mins to copy a directory of size 1.4G. During this time >> window, the searches are slow and the above thread stacks are observed. >> >> >> Could there be any system level limits we're hitting? >> >> >> Our test environment is: >> lucene-2.3.2 >> 4x2.6 GHz, 16G memory >> Red Hat 3.4.6-9 >> >> >> Thanks and regards, >> >> >> - Gagan >> > > > > --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
