Hello, I have been running some experiments on lucene. To speed up index time, I have disabled autocommit, and I flush the indexwriter each 512 objects. So far I have tried with 256,512,1024,and 2048 and I have seen a really incredible speed difference indexing.
However, if I the time required to run optimize() at the end has increased too. So I have been running some more experiments to optimize the index every 512*4, 512*8, etc. but I have not been able to find a sweet spot. Could someone give me some insight on the optimize. Should I run it incrementally while indexing or should I run it at the end. If I run it incrementally, is it better to run it often or not? Best, -C.B.
