Re: indexing performance problems

2009-06-10 Thread Michael McCandless
Thanks for bringing closure! Mike On Wed, Jun 10, 2009 at 4:42 AM, Mateusz Berezecki wrote: > Hi list! > > I'm forwarding as somehow I did not put the list in the CC but the > answer I think is noteworthy, so here it is. Please remember to use > StringBuffer before blaming lucene ;-) > > Actual t

Re: indexing performance problems

2009-06-10 Thread Mateusz Berezecki
Hi list! I'm forwarding as somehow I did not put the list in the CC but the answer I think is noteworthy, so here it is. Please remember to use StringBuffer before blaming lucene ;-) Actual time consumed by lucene is now ~130 minutes as opposed to 20 hours which is neat. I can do much more passes

Re: indexing performance problems

2009-06-08 Thread Mateusz Berezecki
Hi Michael, Thanks a lot for a hint. I'll test it out in a few hours and get back to you and/or the list. best, Mateusz On Mon, Jun 8, 2009 at 2:13 PM, Michael McCandless wrote: > On Mon, Jun 8, 2009 at 7:54 AM, Mateusz Berezecki wrote: > >> Thanks for a prompt response. > > You're welcome! > >>

Re: indexing performance problems

2009-06-08 Thread Michael McCandless
On Mon, Jun 8, 2009 at 7:54 AM, Mateusz Berezecki wrote: > Thanks for a prompt response. You're welcome! >> A mergeFactor of 150 is way too high; I'd put that back to 10 and see >> if the problem persists.  Also make sure you're using >> autoCommit=false, and try the suggestions here: >> >>    h

Re: indexing performance problems

2009-06-08 Thread Mateusz Berezecki
Hi Michael Thanks for a prompt response. On Mon, Jun 8, 2009 at 1:27 PM, Michael McCandless wrote: > This isn't normal. > > A mergeFactor of 150 is way too high; I'd put that back to 10 and see > if the problem persists.  Also make sure you're using > autoCommit=false, and try the suggestions her

Re: indexing performance problems

2009-06-08 Thread Michael McCandless
This isn't normal. A mergeFactor of 150 is way too high; I'd put that back to 10 and see if the problem persists. Also make sure you're using autoCommit=false, and try the suggestions here: http://wiki.apache.org/lucene-java/ImproveIndexingSpeed You're sure the JRE's heap size is big enough