Hello, First many thanks for getting in touch.
Michael McCandless-2 wrote: > > It looks like merging is running too slowly in your environment, > relative to indexing; all of your indexing threads are stuck wanting > to launch a new merge but there's already the max allowed (1) > concurrent merge running and so IW (intentionally) stalls them. > >>> It's exactly what I just noticed. And 5 indexing thread vs 1 merge >>> doesn't sound fair. I read ConcurrentMergeScheduler source code and found out that if maxMergeCount is reached, incoming threads are forcefully paused, which totally make sense. I should try to raise a little bit the maxMergeCount to be accorded with indexing. However, I still don't understand why the merging take so much time (few minutes to an hour). During that time, the IO are almost non existant and CPU stays low. That's why I suspected an locking issue. Michael McCandless-2 wrote: > > Are you sure you passed 2 for numThreads to ThreadedIndexWriter? I see 5 > threads in your stack traces, I think. >>> I actually passed 5 threads in ThreadedIndexWriter, my mistake. Post >>> edited. Michael McCandless-2 wrote: > > What kind of IO system do you have? >>> slow macbook laptop harddrive. I'm actually just trying to get my grips >>> with this wonderful library before, I hope, being able to use it on a >>> proper system. Michael McCandless-2 wrote: > > Can you setInfoStream in the IW and post the output? Could give more > clues... >>> will post that after my next test Simon Willnauer wrote: > > can you explain your configuration as well as the problem? I see one > thread merging (Runnable state) and three other merge threads that are > waiting on a lock. it seems that CMS has put them into wait state and > your indexing threads are blocking on a stall control (too many > merging threads). is it possible that you are flushing very often ie. > low ram buffer or low maxBufferedDocs so that many segments are > created? > >>> I think the flushing is fine (RAMBufferSizeMB:256mb) My configuration is a macbookpro 2.4ghz/4GBram. Not ideal, but the goal is not to benchmark Lucene. -- View this message in context: http://lucene.472066.n3.nabble.com/Thread-locking-while-merging-ConcurrentMergeScheduler-issue-tp3222427p3222946.html Sent from the Lucene - Java Users mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
