On Wed, Aug 3, 2011 at 1:22 PM, Pierre-Henri Toussaint <pierrehenri.toussa...@gmail.com> wrote:
>> It looks like merging is running too slowly in your environment, >> relative to indexing; all of your indexing threads are stuck wanting >> to launch a new merge but there's already the max allowed (1) >> concurrent merge running and so IW (intentionally) stalls them. >> >>>> It's exactly what I just noticed. And 5 indexing thread vs 1 merge >>>> doesn't sound fair. > I read ConcurrentMergeScheduler source code and found out that if > maxMergeCount is reached, incoming threads are forcefully paused, which > totally make sense. I should try to raise a little bit the maxMergeCount to > be accorded with indexing. Well, the core problem is that merging is running too slowly vs how quickly your indexing threads are generating new segments -- so if you raise maxMergeCount, you'll only cause merging to fall even more behind. > However, I still don't understand why the merging take so much time (few > minutes to an hour). During that time, the IO are almost non existant and > CPU stays low. That's why I suspected an locking issue. Yeah that's strange. It could simply be you are using too many indexing threads (plus laptop drive being slowish) -- since you only have 2 cores (Core 2 Duo) maybe cut back from 5 to 2 threads? >> What kind of IO system do you have? >>>> slow macbook laptop harddrive. I'm actually just trying to get my grips >>>> with this wonderful library before, I hope, being able to use it on a >>>> proper system. This could explain the slow merging. >> Can you setInfoStream in the IW and post the output? Could give more >> clues... >>>> will post that after my next test OK maybe we'll get more insight... Mike McCandless http://blog.mikemccandless.com --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
