Wow. Thanks Erick! So I guess the issue isn't with the test code... I wonder what kind of environmental problem I could have? I am also running on XP with JDK 1.5, Lucene 2.1, default memory and gc... The queries I am running are a bit more complex, and return 0-10,000 hits. When I close and re-open the IndexSearcher, all the memory get's gc'd fine. But if I leave it open it runs out of memory fast...
I run the test program through ant so that I can specify all my java versions, jars, and classpaths without referencing system vars. ----- Original Message ---- From: Erick Erickson <[EMAIL PROTECTED]> To: java-user@lucene.apache.org Sent: Thursday, April 5, 2007 7:04:55 PM Subject: Re: How many Searches is a Searcher Worth? Having to put a counter in and close/open your searcher should not be necessary. I'm afraid I'm not going to be very helpful, because.... I took your test case and made some very minor modifications to make it run in an environment I happen to have lying around (mostly, just instantiated the QueryParser with a field known to be in my index). I generated a list of all the terms in that field in my index for the queries.txt file and let it rip. It executed 1,599,586 queries. Admittedly, they were for one and only one term each. Each search had just a few hits because of the peculiarities of my index, usually less than 5. Just for yucks, I tried instantiating the QueryParser with "", and tacking my index field in front of the terms I read from queries.txt. Still no problem. I'm running Lucene 2.1, JDK 1.5, and running the code from inside Eclipse on a Windows XP box. I think the Eclipse JVM only allocates 128M to processes by default...... So, the only thing that comes to mind is the extremely sticky wicket of the environment you're running in. I wonder if it's possible you're CLASSPATH is getting incompatible jar files somehow? Perhaps some jars are coming from from an older JVM? I'm grasping at straws here..... We have production systems running complex queries for days at a time and are processing far more than 40 queries, with no problems so I'm pretty sure it's not Lucene, but something peculiar to your environment. If you're running a Windows machine, depending on how it was installed, sometimes the java installation process will put files in one of the Windows system directories so it's possible that you're hitting something in there even though you aren't trying.... Kudos for writing that test program, by the way. It sure removed a lot of variables from the situation. Best [EMAIL PROTECTED] On 4/5/07, Craig W Conway <[EMAIL PROTECTED]> wrote: > > So, forgetting the RMI stuff, I put together a test client very similar to > the one in the book "Lucene in Action" page 182. > > The client: > > 1. instantiates a IndexSearcher > 2. loops through queries, searches, prints hit count, saves nothing > > I am only able to run through about 40 searches before I get an > OutOfMemoryException. JDK 1.5 > > Because of this, I have put a counter in my search server to close and > re-open the IndexSearcher after a certian number of searches. But this > shouldn't be necessary right? What's eating up all the memory? > > Source code @ http://urbanmarsupial.com/share/TestLuceneMemory.java > > Any hints would be greatly appreciated! > > Thanks, > > Craig > > ----- Original Message ---- > From: Otis Gospodnetic <[EMAIL PROTECTED]> > To: java-user@lucene.apache.org > Sent: Wednesday, April 4, 2007 10:39:50 AM > Subject: Re: How many Searches is a Searcher Worth? > > No reason that I can think of. What makes you think the problem is with > the IndexSearcher? Maybe it's something else in your code, for instance. > Make sure you have the same version of Java on both ends of the > call. Also, Java 6 made our RMI calls a lot more stable than even 1.5. > > Otis > . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . > Simpy -- http://www.simpy.com/ - Tag - Search - Share > > ----- Original Message ---- > From: Craig W Conway <[EMAIL PROTECTED]> > To: java-user@lucene.apache.org > Sent: Wednesday, April 4, 2007 1:25:27 PM > Subject: How many Searches is a Searcher Worth? > > I am using an RMI architecture for calling a remote service which uses an > IndexSearcher in its own JVM. I am starting the service with the following > provisions for memory allocation and garbage collection: java -server > -Xmx1024m -XX:+UseConcMarkSweepGC -XX:+UseParNewGC > > After about 1000 search calls I start to run out of memory, and I have to > close and re-open the IndexSearcher, loosing any cached data and filters... > Is there any reason why I shouldn't be able to use my IndexSearcher forever, > until I want to close it? > > Thanks! > > Craig > > > > > > > ____________________________________________________________________________________ > Now that's room service! Choose from over 150,000 hotels > in 45,000 destinations on Yahoo! Travel to find your fit. > http://farechase.yahoo.com/promo-generic-14795097 > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > > > > > > > > > > ____________________________________________________________________________________ > No need to miss a message. Get email on-the-go > with Yahoo! Mail for Mobile. Get started. > http://mobile.yahoo.com/mail ____________________________________________________________________________________ Never miss an email again! Yahoo! Toolbar alerts you the instant new Mail arrives. http://tools.search.yahoo.com/toolbar/features/mail/
