Re: Bizarre indexing issue where thousands of files get created

2009-08-27 Thread Micah Jaffe
Unfortunately, there's no concise summarization of the code to post. As a follow-up to the situation, after throwing more RAM at the problem (from 2GB to 10GB of process space) we no longer see OOM errors and also no longer see the IndexWriter create thousands of files; the OOM error did ap

Re: Bizarre indexing issue where thousands of files get created

2009-08-18 Thread Jason Rutherglen
Micah, If you can post some of your code, it may be easier to identify the problem you're experiencing. -J On Tue, Aug 18, 2009 at 9:55 AM, Micah Jaffe wrote: > Hi, thanks for the response!  The (custom) searchers that are falling out of > cache are indeed calling close on their IndexReader in f

Re: Bizarre indexing issue where thousands of files get created

2009-08-18 Thread Micah Jaffe
Hi, thanks for the response! The (custom) searchers that are falling out of cache are indeed calling close on their IndexReader in finalize(); they are not calling close on themselves as that appears to be a no-op when creating an IndexSearcher with a reader. The searchers are just extend

Re: Bizarre indexing issue where thousands of files get created

2009-08-18 Thread Michael McCandless
Are you .close()ing your IndexReaders when they fall out of the MRU cache? Seems like there are two problems... 1) why are you hitting OOMEs? Seems likely you're just doing too much at once can you ask the JRE to get you a heap dump when it hits OOME? 2) Why is IndexWriter creating zillions o