Re: Slow merging after upgrading to 3.5

2012-04-18 Thread Michael McCandless
Super, thanks for bringing closure! Mike McCandless http://blog.mikemccandless.com On Wed, Apr 18, 2012 at 5:33 PM, Ivan Brusic wrote: > Just wanted to circle back and report on our progress. > > We finally applied the settings to our production environment and the > improvements have been dram

Re: Slow merging after upgrading to 3.5

2012-04-18 Thread Ivan Brusic
Just wanted to circle back and report on our progress. We finally applied the settings to our production environment and the improvements have been dramatic. Our indexing time has returned to 2.3 levels. Thanks again, Ivan On Fri, Apr 6, 2012 at 11:36 AM, Michael McCandless wrote: > On Thu, Ap

Re: Slow merging after upgrading to 3.5

2012-04-06 Thread Michael McCandless
On Thu, Apr 5, 2012 at 3:31 PM, Ivan Brusic wrote: > On Thu, Apr 5, 2012 at 11:36 AM, Michael McCandless > wrote: >> I'm assuming this is a "build once and never change" index...?  Else, >> it sounds like you should never run forceMerge... > > Correct. The forceMerge was merely to preserve the p

Re: Slow merging after upgrading to 3.5

2012-04-05 Thread Ivan Brusic
Hi Mike, Response inline: On Thu, Apr 5, 2012 at 11:36 AM, Michael McCandless wrote: > I'm assuming this is a "build once and never change" index...?  Else, > it sounds like you should never run forceMerge... Correct. The forceMerge was merely to preserve the previous 2.3 behavior of using opti

Re: Slow merging after upgrading to 3.5

2012-04-05 Thread Michael McCandless
I'm assuming this is a "build once and never change" index...? Else, it sounds like you should never run forceMerge... To preserve insertion order you just need to use one of the Log*MergePolicy (which you are already doing). Merge factor doesn't affect this... For the fastest way to get to a s

Slow merging after upgrading to 3.5

2012-04-05 Thread Ivan Brusic
I recently migrated a legacy Lucene application from 2.3 to 3.5. The code was filled with numerous custom filter/analyzers/similarites/collectors. Took about a week to convert all the token streams to the new API and removed deprecated classes. Most importantly, there is a collector that enables fa