There is bound to be IO contention. I'm sure iostat will give you a much better picture on it.
-- Anshum Gupta http://ai-cafe.blogspot.com On Mon, Aug 23, 2010 at 3:13 PM, <gag...@graffiti.net> wrote: > Yes, all version directories are on the same disk. iostat output should be > useful. Using rsync is complicated because of legacy reasons - there's a > change impact to manage. > > > Intererstingly, the copy is quite fast (around 30s) when there are no > searches in progress. > > > > Thanks, > - Gagan > > > -----Original Message----- > From: Ian Lea <ian....@gmail.com> > To: java-user@lucene.apache.org > Sent: Mon, Aug 23, 2010 2:10 pm > Subject: Re: slow search threads during a disk copy > > > Well, there still sounds like plenty of scope for IO contention. Are > source and dest directories on the same disk? What does iostat show? > Could you use rsync rather than cp -rp, or something else that only > copies changes? > > > -- > Ian. > > > On Mon, Aug 23, 2010 at 9:20 AM, <gag...@graffiti.net> wrote: > > > > We suspected the same at first, and so modified the flow to do the copy > of the > new version "before" searching is activated on it. So, there are no > searches on > the new version dir at the time of copy - yet we still see slowness during > the > disk copy. > > > > > > Thanks, > > - Gagan > > > > > > > > > > > > -----Original Message----- > > From: Anshum <ansh...@gmail.com> > > To: java-user@lucene.apache.org > > Sent: Mon, Aug 23, 2010 1:17 pm > > Subject: Re: slow search threads during a disk copy > > > > > > Seems like a case of I/O issues. You may be reading content off the index > > while performing searches while the I/O for copy is also happening. > > > > -- > > Anshum Gupta > > http://ai-cafe.blogspot.com > > > > > > On Mon, Aug 23, 2010 at 1:12 PM, <gag...@graffiti.net> wrote: > > > >> > >> Hi all, > >> > >> > >> We're observing search threads slowing down during directory copies > >> performed during updates to the index. The thread dump shows search > threads > >> blocked on a FSDirectory$FSIndexInput$Descriptor instance: > >> > >> > >> > >> "Worker Thread - 12" daemon prio=10 tid=0x082b2400 nid=0x4654 waiting > for > >> monitor entry [0x988ed000..0x988edf30] > >> java.lang.Thread.State: BLOCKED (on object monitor) > >> at > >> > org.apache.lucene.store.FSDirectory$FSIndexInput.readInternal(FSDirectory.java:542) > >> - waiting to lock <0xc79c9900> (a > >> org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor) > >> at > >> > org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:152) > >> at > >> > org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:38) > >> at org.apache.lucene.store.IndexInput.readVInt(IndexInput.java:76) > >> at > >> org.apache.lucene.index.SegmentTermDocs.read(SegmentTermDocs.java:133) > >> at > >> > org.apache.lucene.index.MultiSegmentReader$MultiTermDocs.read(MultiSegmentReader.java:573) > >> at org.apache.lucene.search.TermScorer.next(TermScorer.java:106) > >> at > >> > org.apache.lucene.search.ConjunctionScorer.init(ConjunctionScorer.java:80) > >> at > >> > org.apache.lucene.search.ConjunctionScorer.next(ConjunctionScorer.java:48) > >> at > >> org.apache.lucene.search.BooleanScorer2.score(BooleanScorer2.java:319) > >> at > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:146) > >> at org.apache.lucene.search.Searcher.search(Searcher.java:118) > >> > >> ... > >> > >> > >> The directory copy we do is a 'cp -pr <src-dir>/* <dest-dir>' command > prior > >> to applying changes (addDocument calls) on the current "available" > segment. > >> This takes >7 mins to copy a directory of size 1.4G. During this time > >> window, the searches are slow and the above thread stacks are observed. > >> > >> > >> Could there be any system level limits we're hitting? > >> > >> > >> Our test environment is: > >> lucene-2.3.2 > >> 4x2.6 GHz, 16G memory > >> Red Hat 3.4.6-9 > >> > >> > >> Thanks and regards, > >> > >> > >> - Gagan > >> > > > > > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org > > >
