Re: Concurrency and multiple merge threads

2012-02-19 Thread Mike McCandless
Sounds like a nice machine! It's frustrating that RAMFile even has any sync'd methods... Lucene is write once, so once a RAMFile is written we don't need any sync to read it. Maybe on creating a RAMInputStream we could make a new ReadOnlyRAMFile, holding the same buffers without sync. That sa

Re: Concurrency and multiple merge threads

2012-02-18 Thread Greg Bowyer
Your not very clear about where you see the specific slow operations, at search or re-index time. I am going to go out on a limb here and suggest that maybe its at index time, and maybe the yourkit trace showing the 5 merge threads awaiting the monitor is the cause of your issues. You claim

Concurrency and multiple merge threads

2012-02-18 Thread Benson Margulies
Using Lucene 3.5.0, on a 32-core machine, I have coded something shaped like: make a writer on a RAMDirectory. start: Create a near-real-time searcher from it. farm work out to multiple threads, each of which performs a search and retrieves some docs. When all are done, write some new do