Hi Magnus, Could you post the OS, version, RAM size, swapsize, Java VM version, hardware, #cores, VM command line parameters, etc? This can be very relevant.
Have you tried other garbage collectors and/or tuning as described in http://java.sun.com/javase/technologies/hotspot/gc/gc_tuning_6.html? 2008/12/3 Magnus Rundberget <[EMAIL PROTECTED]>: > Hi, > > We have an application using Tomcat, Spring etc and Lucene 2.4.0. > Our index is about 100MB (in test) and has about 20 indexed fields. > > Performance is pretty good, but we are experiencing a very high usage of > memory when searching. > > Looking at JConsole during a somewhat silly scenario (but illustrates the > problem); > (Allocated 512 MB Min heap space, max 1024) > > 0. Initially memory usage is about 70MB > 1. Search for word "er", heap memory usage goes up by 100-150MB > 1.1 Wait for 30 seconds... memory usage stays the same (ie no gc) > 2. Search by word "og", heap memory usage goes up another 50-100MB > 2.1 See 1.1 > > ...and so on until it seems to reach the 512 MB limit, and then a garbage > collection is performed > i.e garbage collection doesn't seem to occur until it "hits the roof" > > We believe the scenario is similar in production, were our heap space is > limited to 1.5 GB. > > > Our search is basically as follows > ---------------------------------------------- > 1. Open an IndexSearcher > 2. Build a Boolean Query searching across 4 fields (title, summary, content > and daterangestring YYYYMMDD) > 2.1 Sort on title > 3. Perform search > 4. Iterate over hits to build a set of custom result objects (pretty small, > as we dont include content in these) > 5. Close searcher > 6. Return result objects. You should not close the searcher: it can be shared by all queries. What happens when you warm Lucene with a (large) number of queries: do things stabilize over time? A 100MB index is (relatively) very small for Lucene (I have indexes >100GB). What kind of response times are you getting, independent of memory usage. -glen > > We have tried various options based on entries on this mailing list; > a) Cache the IndexSearcher - Same results > b) Remove sorting - Same result > c) In point 4 only iterating over a limited amount of hits rather than whole > collection - Same result in terms of memory usage, but obviously increased > performance > d) Using RamDirectory vs FSDirectory - Same result only initial heap usage > is higher using ramdirectory (in conjuction with cached indexsearcher) > > > Doing some profiling using YourKit shows a huge number of char[], int[] and > string[], and ever increasing number of lucene related objects. > > > > Reading through the mailing lists, suspicions are that our problem is > related to ThreadLocals and memory not being released. Noticed that there > was a related patch for this in 2.4.0, but it doesn't seem to help us much. > > Any ideas ? > > kind regards > Magnus > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > -- - --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
