Oh yes, I forget to mention that MaxPermSize is very useful, and may be another key that solves my problem. I haven't tried UseConcMarkSweepGC and the other two parameters, and I will try them instead of my own GC thread to see whether the problem can also be solved.
Thanks Brian! Regards, Gong > -----Original Message----- > From: Beard, Brian [mailto:[EMAIL PROTECTED] > Sent: Monday, October 06, 2008 8:48 PM > To: java-user@lucene.apache.org > Subject: RE: Memory eaten up by String, Term and TermInfo? > > I played around with GC quite a bit in our app and found the following > java settings to help a lot (Used with jboss, but should be > good for any > jvm). > > set JAVA_OPTS=%JAVA_OPTS% -XX:MaxPermSize=512M -XX:+UseConcMarkSweepGC > -XX:+CMSPermGenSweepingEnabled -XX:+CMSClassUnloadingEnabled > > While these settings did help when the GC starts to get taxed for > keeping throughput better, the biggest thing was getting the heap size > big enough. If the heap's big enough, then everything seems > to work well > regardless of the GC algorithm selected. > > If the heap size and algorithm are correct, then you shouldn't have to > force GC. Usually the forced GC calls will trigger the total GC which > has a long delay and slows down responsiveness to the app. > > I found jstat fairly helpful in monitoring all of this. You can see if > the following article helps at all. > > http://java.sun.com/javase/technologies/hotspot/gc/index.jsp > > -----Original Message----- > From: Peter Cheng [mailto:[EMAIL PROTECTED] > Sent: Sunday, October 05, 2008 7:55 AM > To: java-user@lucene.apache.org > Subject: RE: Memory eaten up by String, Term and TermInfo? > > I have confirmed that the OutOfMemoryError is not Lucene's > problem. It's > just because JVM failed to perform GC when necessary, and I don't know > why. > To fix this, I started another thread to call GC every six hours, and > problems got solved. > > Thank you all. > > Gong > > > -----Original Message----- > > From: Michael McCandless [mailto:[EMAIL PROTECTED] > > Sent: Sunday, September 14, 2008 10:28 PM > > To: java-user@lucene.apache.org > > Subject: Re: Memory eaten up by String, Term and TermInfo? > > > > > > Small correction: it was checked in this morning (at least, on the > > East Coast of the US). > > > > So you need to either build your own JAR using Lucene's trunk, or, > > wait for tonite's build to run and then download the build > artifacts > > from here: > > > > http://hudson.zones.apache.org/hudson/job/Lucene-trunk > > > > If possible, please report back if this fixed your OutOfMemoryError. > > > > 2.4 will include this fix. > > > > Mike > > > > Chris Lu wrote: > > > > > Can you try to update to the latest Lucene svn version, like > > > yesterday? > > > LUCENE-1383 was checked in yesterday. This patch is > > addressing a leak > > > problem particular to J2EE applications. > > > > > > -- > > > Chris Lu > > > ------------------------- > > > Instant Scalable Full-Text Search On Any Database/Application > > > site: http://www.dbsight.net > > > demo: http://search.dbsight.com > > > Lucene Database Search in 3 minutes: > > > > > http://wiki.dbsight.com/index.php?title=Create_Lucene_Database > > _Search_in_3_minutes > > > DBSight customer, a shopping comparison site, (anonymous per > > > request) got > > > 2.6 Million Euro funding! > > > > > > On Sun, Sep 14, 2008 at 6:58 AM, Peter Cheng > > > <[EMAIL PROTECTED]>wrote: > > > > > >> Hi the community, > > >> > > >> In a Tomcat application (a keyword-based search engine), I use > > >> Lucene 2.3.2 > > >> to index 60 million documents, but each document is > small-sized. > > >> The total > > >> index size is about 60GB. > > >> After a successful running for a week, Tomcat was down due to > > >> OutOfMemory. > > >> Then I restarted Tomcat, and after three days, I used jmap > > and jhat > > >> to find > > >> out what had eaten up the memory. I found millions of > > instances of > > >> String, > > >> Term, and TermInfo. Why? > > >> > > >> In my application, I use a single IndexSearcher object, > which is > > >> shared by > > >> all the requests. It is opened initially, and will never > be closed. > > >> > > >> What could have eaten up the memory? What is referring to > > millions of > > >> instances of Term and TermInfo? > > >> > > >> I can provide any snippets of codes if necessary. > > >> Thank you so much! > > >> > > >> Gong Cheng > > >> > > >> > > >> > > > --------------------------------------------------------------------- > > >> To unsubscribe, e-mail: [EMAIL PROTECTED] > > >> For additional commands, e-mail: [EMAIL PROTECTED] > > >> > > >> > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [EMAIL PROTECTED] > > For additional commands, e-mail: [EMAIL PROTECTED] > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
