Re: HeapedScorerDoc using all my memory

2009-04-03 Thread Michael McCandless
I'm also confused, because ScorerDocQueue should not be used during indexing. It's used only when scoring boolean "OR" queries. Are you doing searching in the same JVM as indexing? Mike On Fri, Apr 3, 2009 at 9:21 AM, John Byrne wrote: > Unfortunately I'm not sure of the exact number. It happe

Re: HeapedScorerDoc using all my memory

2009-04-03 Thread Erick Erickson
Good Luck! I love (a little sarcasm there) being presented with problem statements like "it doesn't work. You can't do anything on the machine where the problem is. We can't give you any information about what's happening. How long will it take you to fix it?" Best er...@infullsympathy.com On Fr

Re: HeapedScorerDoc using all my memory

2009-04-03 Thread John Byrne
Unfortunately I'm not sure of the exact number. It happened on a machine I have no access to, and I was just e-mailed a few details of the problem! We have a JMS queue, where each message is a file to be indexed. There was somewhere between 2000 and 10,000 messages processed when it happened.

Re: HeapedScorerDoc using all my memory

2009-04-03 Thread Erick Erickson
H, that's odd. how many is "a large number of documents"? And what is your index size when things to wonky? (approximately) I can say that other people create very large indexes without this happening, but the only thing that says is that this isn't a *known* problem. Is there any chance you'

Re: HeapedScorerDoc using all my memory

2009-04-03 Thread John Byrne
The maximum JVM memory is 2GB. Apparently 1.2GB is being used up by this class. All IndexWriter settings are left as default. I haven't tried any changes yet, because the problem so far has on ly happened in a production environment that I can't mess with. I am planning to try reproducing it

Re: HeapedScorerDoc using all my memory

2009-04-03 Thread Erick Erickson
How much memory are you allocating for the JVM? And what are your various indexwriter settings (e.g. MaxBufferedDocs, MaxMergeDocs, etc). Have you tried different settings in setRamBufferSizeMB? Best Erick On Fri, Apr 3, 2009 at 7:13 AM, John Byrne wrote: > Hi, I'm having a problem where the J