Hi, we are using a search system based on Lucene and have recently tried to add incremental updating of the index instead of building a new index every now and then. However we now run into problems as our searches starts to take very long time to complete.
Our index is about 8-9GB large and we are sending lots of updates / second (we are probably merging in 200 - 300 in a few seconds). Today we buffer a bunch of updates and then merge them into the existing index like a batch, first doing deletes and then inserts. We are currently not using any special tuning of Lucene. Does anyone have any similiar experiences from Lucene or advices on how to reduce the amount of times it takes to perform a search? In particular what would be an optimal combination of update size, merge factor, max buffered docs? /Rickard
