Or you can just buffer your update requests and delete in batch and then add in batch.
Or you could keep that IndexReader on the large index and use it to delete objects, while doing adds on a RAMDirectory. Then, when you are done, merge the RAMDirectory index with the large one. Also just an untested idea.... and really something that is not much different from simple buffering in your application logic. Otis --- Volodymyr Bychkoviak <[EMAIL PROTECTED]> wrote: > Hi, > > just an idea how to manage large index that is updated very often. > > Very often there is need to update an document in index. To update > document in index you should delete old document from index and then > add > new one. In most cases it require you to open IndexReader, delete > document, close IndexReader, create IndexWriter, add document, close > IndexWriter, and re-open IndexSearcher (if index is searched > heavily). > Profiling some applications I found that most time is spend in > IndexReader.open() method. Also it produces many objects, so it also > gives GC overhead. > > Idea to optimize this process is to create two indexes. One main > index > that could be very large and second index that will serve as "change > buffer". We can keep one IndexReader open for the first index. (and > use > it for searching and for deleting old documents). Second index is > small > and we can reopen IndexReader frequently when needed. > > when second index reaches some number of documents we can merge it > with > main index. > to search this "multi" index we could use MultiSearcher over this two > > indexes but with little trick: first IndexSearcher is kept same > during > all time till second index is merged with main and second > IndexSearcher > is reopened when second index changes. > > It is just idea. (It is not tested) > > Will it help to improve speed of updating large index and lower > memory > overhead? > Any comments? > > Regards, > Volodymyr Bychkoviak > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
