On Friday 09 January 2009 05:29:15 John Wang wrote:
> Makes sense.
> I didn't think 32 was the empirically determined magic number ;)

That number does have a history, but I don't know the details.
 
> Are you planning to do a patch for this?

No, but could you open an issue and mention the performance
improvements?

Regards,
Paul Elschot


> 
> -John
> 
> On Thu, Jan 8, 2009 at 1:27 AM, Paul Elschot <paul.elsc...@xs4all.nl> wrote:
> 
> > John,
> >
> > Continuing, see below.
> >
> > On Wednesday 07 January 2009 14:24:15 Paul Elschot wrote:
> > > On Wednesday 07 January 2009 07:25:17 John Wang wrote:
> > > > Hi:
> > > >
> > > >    The default buffer size (for docid,score etc) is 32 in TermScorer.
> > > >
> > > >     We have a large index with some terms to have very dense doc sets.
> > By
> > > > increasing the buffer size we see very dramatic performance
> > improvements.
> > > >
> > > >     With our index (may not be typical), here are some numbers with
> > buffer
> > > > size w.r.t. performance in our query (a large OR query):
> > > >
> > > >     Buffer-size  improvement
> > > > 2042 -       22.0 %
> > > > 4084 -       39.1 %
> > > > 8172 -       51.1 %
> > > >
> > > >     I understand this may not be suitable for every application, so do
> > you
> > > > think it makes sense to make this buffer size configurable?
> > > >
> > >
> > > Ideally the TermScorer buffer size could be set to a size depending on
> > > the query structure, but there is no facility for this yet.
> > > For OR queries larger buffers help, but not for AND queries.
> > > See also LUCENE-430 on reducing buffer sizes for the underlying
> > > TermDocs for very sparse doc sets.
> >
> > It may be possible to change the TermScorer buffer size dynamically.
> > For OR queries TermScorer.next() is used, and for AND queries
> > TermScorer.skipTo() is used.
> > That means that when the buffer runs out during TermScorer.next(),
> > it could be enlarged, for example by doubling (or quadrupling) the size
> > to a configurable maximum of 8K or even 16K, see above. When
> > TermScorer.skipTo() runs out of the buffer it could leave the buffer
> > size unchanged.
> >
> > This involves some memory allocation during search.
> > That is unusual, but it could be worthwhile given the
> > performance improvement.
> >
> > Regards,
> > Paul Elschot
> >
> 

Reply via email to