Thanks for your reply MIke, I create this bug. https://issues.apache.org/jira/browse/LUCENE-3841
Matt On Thu, Mar 1, 2012 at 2:32 PM, Michael McCandless <luc...@mikemccandless.com> wrote: > Phew, tricky. > > The problem is purging is potentially costly... it iterates all > entries in the map (threads that have called get) looking for dead > threads. > > Can you open an issue...? We can iterate there. Thanks for raising this, > > Mike McCandless > > http://blog.mikemccandless.com > > On Wed, Feb 29, 2012 at 12:17 PM, Matthew Bellew <matth...@labkey.com> wrote: >> We tracked down a large memory leak (effectively a leak anyway) caused >> by how Analyzer users CloseableThreadLocal. >> CloseableThreadLocal.hardRefs holds references to Thread objects as >> keys. The problem is that it only frees these references in the set() >> method, and SnowballAnalyzer will only call set() when it is used by a >> NEW thread. >> >> The problem scenario is as follows: >> >> The server experiences a spike in usage (say by robots or whatever) >> and many threads are created and referenced by >> CloseableThreadLocal.hardRefs. The server quiesces and lets many of >> these threads expire normally. Now we have a smaller, but adequate >> thread pool. So CloseableThreadLocal.set() may not be called by >> SnowBallAnalyzer (via Analyzer) for a _long_ time. The purge code is >> never called, and these threads along with their thread local storage >> (lucene related or not) is never cleaned up. >> >> I think calling the purge code in both get() and set() would have >> avoided this problem, perhaps using WeakHashMap instead of HashMap may >> also have helped (WeakHashMap purges on get() and set()) >> >> Our current work around is to not share SnowBallAnalyzer instances >> among HTTP searcher threads. We open and close one on every request. >> >> Thanks, >> Matt >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >> For additional commands, e-mail: java-user-h...@lucene.apache.org >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
