On Thu, Jan 3, 2013 at 12:16 PM, Alon Muchnick <a...@datonics.com> wrote: > hi Mike , > > at the peak there are 500 live threads going through Lucune (not all of > them at the same time , tomcat thread pool uses round robin ) ,regarding > the Directory impl we are using RAMDirectory. > the object that takes most of the heap is the "hardRefs" WeakHashMap class > member in the CloseableThreadLocal class . the size of the map is 500 , > with one entry for each thread that went through Lucune.
Hmm ... I'm curious what's actually using up all the RAM here. A buffered Directory impl (eg NIOFSDirectory) would have IndexInput clones that have 1 KB buffers, but RAMDirectory doesn't do that. > when in run the test with only one thread the initial RAM did not grow much > beyond the initial 30MB . OK that's good. So somehow each thread ties up ~ 1 MB RAM. > when looking at the code i can see that the references to the hardRefs map > will be cleared when : > > 1.a thread which searched Lucune is no longer alive , so its corresponding > record in the map will be cleared either by the GC or by the purge() method > . > > 2.the close() method is called which "should only be called when all > threads are done using the instance" . Right. > what happens to threads that go through Lucune and stay alive for a very > long times ? > will they always be a reference key in the hardRefs map for them ? Yes ... we never prune live threads. > if so does the value for the corresponding record in the map will be > overwritten each time this thread makes a new search ? or will it be some > how accumulated ? : Well, we set the value once when we first see the thread, and then use that value from then on. > value ---- org.apache.lucene.index.TermInfosReader$ThreadResources ---> > > termInfoCache |org.apache.lucene.util.cache.SimpleLRUCache > termEnum |org.apache.lucene.index.SegmentTermEnum > > i have reduced the size of the thread pool , and changed to a more > aggressive thread closing policy ,threads are now reduced when they are not > needed (before that the thread pool would reach its max size and stayed > there), > this should kick in the the purge method and reclaim memory from the closed > threads . > > ill give the system a few hours to run and update on the results . OK thanks. Really anything more then 2 * number-of-cores threads coming through Lucene is overkill ... Mike McCandless http://blog.mikemccandless.com --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
