Batching the updates really ought to improve overall throughput. Have you tried with even bigger batches (100,1000 docs)?
But, how large is each update? Are you changing any IndexWriter settings, e.g. ramBufferSizeMB. Using threads should help too, at least a separate thread doing indexing from calling SearcherManager.maybeRefresh (and, separate threads doing searching). You can also check out http://blog.mikemccandless.com/2011/06/lucenes-near-real-time-search-is-fast.htmlwhere I go into some detail on speeding up indexing rate and refresh speed with near-real-time ... Mike McCandless http://blog.mikemccandless.com On Thu, Apr 25, 2013 at 11:10 PM, Aleksey <bitterc...@gmail.com> wrote: > Hey guys, > > I'm new to Lucene and I was trying to estimate how fast I can make updates > to the index and reopen it. The behavior I'm seeing seems odd. > I'm using Lucene4.2 and SearchManager instance that takes an index writer. > > I make a loop where I update 1 document, then call maybeRefresh and acquire > new searcher and make a search to verify that the update is there. On my > laptop this does about 100 iterations per second. > > Then I run another loop but make 10 updates before reopening the index, and > this only does 10 iterations per second, proportionally less. I was > expecting that if I batch the updates, I can get higher overall throughput, > but that does not seem to be the case. The size of the index I'm updating > doesn't make a difference either, I tried 3K and 100K document sets, 1-2K > each doc, but both produce same update speed (though I'm not calling commit > in these instances). > > Can anyone point me into the right direction to investigate this or hint > how to maximize write throughput to the index, while still <.5 second > delays in seeing the updates. > > Thank you in advance, > > Aleksey >
