Hi Jake
Thanks for the info. Are you specifically referring to http://issues.apache.org/jira/browse/LUCENE-2120? Our app indexes about 170 50k documents per second in heavy load. In any case, I doubt it is the realtime IndexReader issue since when we ran the test, no searching was taking place.
We have been using Lucene search on a real time basis for years and have not experienced any problems until now. Thanks for the tip on Zoie, but we cannot use any additional frameworks on top of Lucene
as Lucene is now deeply integrated into our app. Regards, Jamie On 2010/01/27 08:42 AM, Jake Mannix wrote:
Hi Jamie, How fast are you indexing (number of documents per second)? We also ran into this when trying to perf test heavy query throughput while doing rapid indexing under exactly these conditions: call getReader() every time a search is executed (so that it's "really real time"). The answer is that calling getReader() on every search request is really not a supported operation under heavy indexing load, currently. You are advised to cache the reader you get from this call, and only refresh once per some interval of time (determined by your need: if you need real-time up to every 5 seconds, refresh every 5s. If you need every second, refresh every second, etc). If you really want to have the functionality of being able to get a completely fresh view of the index on each search, look at the Zoie project ( http://zoie.googlecode.com ), an Apache-licensed realtime search system built on top of Lucene which we open-sourced out of LinkedIn a couple of years ago. Zoie allows for immediate reopen on request, and handles enormously heavy indexing and query load (I recently demo'ed zoie to the guys at Twitter at a tech-talk, and showed off indexing about 1500 tweets per second (it could have done more, but it was virtualized disk - EC2) while slamming it with full-throttle query throughput, all while reopening for each request, and it didn't fall over). If you want to try it out, drop me a line and I can help if you need any. -jake
--------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
