Re: Index Merging Space Requirements

2008-03-13 Thread Michael McCandless
Well ... yes and no? Yes, the Log*MergePolicy will still at certain times merge the index all the way down to one segment. If mergeFactor is 10 then this will happen every "power of 10" flushed segments. Ie, after 10 flushes a merge will merge them down to 1 segment, then after 100 flush

Re: Index Merging Space Requirements

2008-03-13 Thread Mark Miller
Thanks a lot Mike...one more question: I remember reading that a regular addDocument call could basically trigger an optimize on a given call. Is this true? Maybe not true anymore? It doesnt sound right to me, but I do remember reading about it. This was pre background merging when it was men

Re: Index Merging Space Requirements

2008-03-13 Thread Michael McCandless
Yes this should reduce transient (while merging) disk usage. However, optimize disregards this parameter, so it will still use the same disk space. However, if you call optimize(N) then that should use less space since it does not merge all the way down to 1 segment. Note that the limit

Index Merging Space Requirements

2008-03-13 Thread Mark Miller
If I use LogByteSizeMergePolicy#setMaxMergeMB, can I clamp down on the space needed for optimize/merge? My thought is, if a segment is maxed out, it will never need to be copied for a merge right? So you could significantly reduce merge/optimize space requirments (now at like 2x-4x if readers c