Re: Incremental updates / slow searches.

2006-10-11 Thread Rickard Bäckman
Thanks for the suggestions. We tried to reduce the amount of times we open a new searcher with some progress. However a lot of our searches still times out. We are currently opening a new searcher and warms it up before doing the switch. We even map the fields we are using for deleting to the doc

Re: Incremental updates / slow searches.

2006-10-09 Thread Mathias Lux
Rickard Bäckman wrote: > Hi, > > we are using a search system based on Lucene and have recently tried to add > incremental updating of the index instead of building a new index every now > and then. However we now run into problems as our searches starts to take > very long time to complete. > >

Re: Incremental updates / slow searches.

2006-10-09 Thread Yonik Seeley
On 10/9/06, Chris Hostetter <[EMAIL PROTECTED]> wrote: don't forget to optimize your index every now and then as well... deleting a document just marks it as "deleted" it still gets inspectected by every query during scoring at least once to see that it can skip it, optimizing is the only thing t

Re: Incremental updates / slow searches.

2006-10-09 Thread Chris Hostetter
quot; documents. : Date: Mon, 9 Oct 2006 13:49:34 -0400 : From: Yonik Seeley <[EMAIL PROTECTED]> : Reply-To: java-user@lucene.apache.org : To: java-user@lucene.apache.org : Subject: Re: Incremental updates / slow searches. : : The biggest thing would be to limit how often you open a new : Index

Re: Incremental updates / slow searches.

2006-10-09 Thread Yonik Seeley
The biggest thing would be to limit how often you open a new IndexSearcher, and when you do, warm up the new searcher in the background while you continue serving searches with the existing searcher. This is the strategy that Solr uses. There is also the issue of if you are analyzing/merging doc