Doron wrote:
> Not sure if this is the case, but you said "searchers", so might be it -
> you can (and should) reuse searchers for multiple/concurrent queries.
> IndexSearcher is thread-safe, so no need to have a different searcher for
> each query. Keep using this searcher until you decide to open a new
> searcher - actually until you 'warmed up' the new (updated) searcher,
> then switch to using the new searcher and close the old one.
>
> - Doron
Thanks Doron, yes we are re-using our Searchers. The particular piece of
code I refer to below, written by Peter Halcsy contains a method called
'getSearcher()' and this returns a cached IndexSearcher. However first
it checks whether the index has been changed since the current Searcher
was instantiated, and if so, 'retires' the current, cached Searcher and
returns a brand new one. Any retired Searchers get removed once all their
current queries are finished.
I have re-worked that piece to put a configurable cap on the number of
these, which should stop the open-ended behaviour which would lead to
our OOM's. Once that max is reached, getSearcher() keeps returning
the current cached Searcher whether the index is changed or not.
I've tested this and seen it in action by continuously indexing docs at
about 40ms per document, and writing a query benchmarker client which
forks N times, with each child looping through a bunch of pre-prepared
queries. Together with some good logging I can see the searchers being
created up to the max. number, then as I vary the indexing throughput
it drops back and new Searchers are once more created to get searching
back up to date with the index changes.
Cheers,
Paul.
> Paul Waite <[EMAIL PROTECTED]> wrote on 18/10/2006 18:22:30:
>
> > Some excellent feedback guys - thanks heaps.
> >
> >
> > On my OOM issue, I think Hoss has nailed it here:
> >
> > > That said: if you are seeing OOM errors when you sort by a field (but
> > > not when you use the docId ordering, or sort by score) then it sounds
> > > like you are keeping refrences to IndexReaders arround after you've
> > > stoped using them -- the FieldCache is kept in a WeakHashMap keyed
off
> of
> > > hte IndexReader, so it should get garbage collected as soon sa you
let
> go
> > > of it. Another possibility is that you are sorting on too many
fields
> > > for it to be able to build the FieldCache for all of them in the RAM
> you
> > > have available.
> >
> >
> > I'm using a piece of code written by Peter Halacsy which implements a
> > SearcherCache class. When we do a search we request a searcher, and
this
> > class looks after giving us one.
> >
> > It checks whether the index has been updated since the most recent
> Searcher
> > was created. If so it creates a new one.
> >
> > At the same time it 'retires' outdated Searchers, once they have no
> queries
> > busy with them.
> >
> > Looking at that code, if the system gets busy indexing new stuff, and
> doing
> > complex searches this is all rather open-ended as to the potential
number
> > of fresh Searchers being created, each with the overhead of building
its
> > FieldCache for the first time. No wonder I'm having problems as the
> archive
> > has grown! Looking at it in this light, my OOM's all seem to come just
> > after a bout of articles have been indexed, and querying is being done
> > simultaneously, so it does fit.
> >
> > I guess a solution is probably to cap this process with a maximum
number
> > of active Searchers, meaning potentially some queries might be fobbed
off
> > with slightly out of date versions of the index, in extremis, but it
> would
> > right itself once everything settles down again.
> >
> > Obviously the index partitioning would probably make this a non-issue,
> but
> > it seems better to sort the basic problem out anyway, and make it 100%
> > stable.
> >
> > Thanks Hoss!
> >
> > Cheers,
> > Paul.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [EMAIL PROTECTED]
> > For additional commands, e-mail: [EMAIL PROTECTED]
> >
--
Suspicion always haunts the guilty mind.
-- Wm. Shakespeare
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
