Lucene's indexing is very concurrent, especially in 4.x where we've added concurrent flushing: http://blog.mikemccandless.com/2011/05/265-indexing-speedup-with-lucenes.html
Are you sure the bottleneck is in Lucene and not e.g. the server-side handling of suddenly receiving 110 X 10,50,100 documents? E.g. perhaps most of your requests are queueing up, if the server only allows N requests at once. If you make a standalone Lucene-only test indexing the same sudden surge of documents, how long does it take? How many threads actually interact with Lucene? Mike McCandless http://blog.mikemccandless.com On Tue, Feb 11, 2014 at 12:25 AM, Umashanker, Srividhya <srividhya.umashan...@hp.com> wrote: > Group - > > > > We have an Indexing and Searching Service (using Lucene 4.0) implemented over > REST as part of our framework, which all the related modules will use to > publish data that make it available for the UI. > > Moreover, every rest call that our service receives has a proxy timeout limit > of 200 ms. > > Initially we started concurrency tests with Indexing. Recently, we started > doing some concurrency tests on our service to handle min 110 users, who > will be publishing json data ranging from 10, 50 100 documents by one user. > The document is not huge like what u are using for your performance tests. > Mostly we call updateDocument() as we do not know whether the document is > already indexed or not > > > > 110 users/threads with a ramp up time of 0 seconds seems to fail randomly > because some of the threads/REST call do not complete within 200 ms. > > > > We tried the following. > > 1. maxBufferSize of 500 MB. > > 2. DWPT count to 10. > > 3. Tried updateDocument in separate thread, as said in one of your > suggestions in the internet. Doesn't seem to help. > > 4. TiredMergePolicy has been set. Unfortunately this does not help as the > maxBufferSize is not hit while we index, and merge does not happen yet. > > 5. We are yet to try with NMapDirectory. > > > > Are there any other suggestions/experiences that you can share with us. Are > there any benchmark tests that you have done on concurrency? > > > > Please help us. > > > > -Vidhya > > --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
