my maxBufferedDocs is 1000, do you recommend bigger than that? What's a good number for this for a very high indexing rate (10K new documents every min).
On Jan 19, 2008 10:30 PM, Otis Gospodnetic <[EMAIL PROTECTED]> wrote: > In addition to what Mike already said: > > maxMergeDocs=99999 -- do you really mean maxMergeDocs and not maxBufferedDocs? > > Larg(er) maxBufferedDocs will speed up indexing. > > Otis > > -- > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch > > ----- Original Message ---- > From: vivek sar <[EMAIL PROTECTED]> > To: java-user@lucene.apache.org > Sent: Friday, January 18, 2008 4:31:26 AM > Subject: Optimize for large index size > > Hi, > > We are using Lucene 2.2. We have an index of size 70G (within 3-4 > days) and growing. We run optimize pretty frequently (once every hour > - due to large number of index updates every min - can be up to 100K > new documents every min). I have seen every now and then the optimize > takes 3-4 hours to complete and up to 8 G memory (our limit). This > makes the whole system slow. Few questions, > > 1) Is there any alternative to optimize? That is, can we do without > optimize and still have our search fast? > 2) What's the best way to use optimize, i.e. how can we make the > optimize much faster and use lesser memory? > 3) Is there a way to partition the indexes using Lucene? Let's say we > partition daily, so we have to optimize only the daily indexes and not > the whole thing. > > Our mergefactor=200 and maxMergeDocs=99999 > > Thanks, > -vivek > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
