Re: ThreadLocal in SegmentReader

2008-07-11 Thread Roman Puchkovskiy
Well, this 'replacement' of the ThreadLocal does not solve the initial problem. As there's always at least one ThreadLocal which binds the object loaded by the web-app to the Thread which is _not_ loaded by the web-app, the classloader never may be unloaded. You are right, this is not the 'leak'

Re: ThreadLocal in SegmentReader

2008-07-11 Thread Michael McCandless
After discussing this on java-dev: http://mail-archives.apache.org/mod_mbox/lucene-java-dev/200807.mbox/[EMAIL PROTECTED] it seems that this is not in fact a leak but rather a delayed GC issue. The objects are eventually freed, on Sun 1.4, 1.5 and 1.6. When a ThreadLocal instance beco

Re: ThreadLocal in SegmentReader

2008-07-07 Thread Michael McCandless
Ugh! I'll move this to java-dev to brainstorm fixes... Mike Yonik Seeley wrote: On Mon, Jul 7, 2008 at 2:43 PM, Michael McCandless <[EMAIL PROTECTED]> wrote: So now I'm confused: the SegmentReader itself should no longer be reachable, assuming you are not holding any references to your In

Re: ThreadLocal in SegmentReader

2008-07-07 Thread Yonik Seeley
On Mon, Jul 7, 2008 at 2:43 PM, Michael McCandless <[EMAIL PROTECTED]> wrote: > So now I'm confused: the SegmentReader itself should no longer be reachable, > assuming you are not holding any references to your IndexReader. > > Which means the ThreadLocal instance should no longer be reachable. It

Re: ThreadLocal in SegmentReader

2008-07-07 Thread Michael McCandless
So now I'm confused: the SegmentReader itself should no longer be reachable, assuming you are not holding any references to your IndexReader. Which means the ThreadLocal instance should no longer be reachable. Which means it should be GC'd and everything it's holding should be GC'd as we

Re: ThreadLocal in SegmentReader

2008-07-07 Thread Roman Puchkovskiy
I've tested a little, and it seems that assigning a null is not sufficient. As expected... I don't see other ways how to fix this, but I'm not the Lucene developer :) Fortunately, there's the work-around with temporary thread. Michael McCandless-2 wrote: > > > Hmmm, I see... you're right. Thr

Re: ThreadLocal in SegmentReader

2008-07-07 Thread Roman Puchkovskiy
Yes, calling set(null) does not seem a good fix. As for setting a reference to termVectorsLocal to null, not sure could this help or not, as this ThreadLocal will still be referenced by the thread (or threads). Anyway, I will try to test this approach and post the results here. Michael McCandles

Re: ThreadLocal in SegmentReader

2008-07-07 Thread Michael McCandless
Hmmm, I see... you're right. ThreadLocal is dangerous. So how would you recommend fixing it? One thing we can do, in SegmentReader.close, is to call termVectorsLocal.set(null). We do this eg in FieldsReader.close, which uses a ThreadLocal to hold thread-private clones of the fieldsStrea

Re: ThreadLocal in SegmentReader

2008-07-07 Thread Roman Puchkovskiy
Unfortunately, it's not ok sometimes. For instance, when Lucene is loaded by a web-application from its WEB-INF/lib and SegmentReader is initialized during the application start-up (i.e. it's initialized in the thread which will never die), this causes problems with unloading of a classloader of t

Re: ThreadLocal in SegmentReader

2008-07-07 Thread Michael McCandless
Well ... if the thread dies, the value in its ThreadLocal should be GC'd. If the thread does not die (eg thread pool in an app server) then the ThreadLocal value remains, but that value is a shallow clone of the original TermVectorsReader and should not be consuming that much RAM per th