Unfortunately I'm not sure of the exact number. It happened on a machine
I have no access to, and I was just e-mailed a few details of the
problem! We have a JMS queue, where each message is a file to be
indexed. There was somewhere between 2000 and 10,000 messages processed
when it happened.
Because we don't make a distinction between batch and incremental
indexing, I am closing and re-opening the writer for each document.
What you've suggested there could well be the problem, that's certainly
where I'll look next.
Thanks for the ideas anyway - I know I really need to come up with some
more info on the problem, so I think the next thing I'll do it try to
reproduce it locally.
-John
Erick Erickson wrote:
Hmmmm, that's odd. how many is "a large number of documents"? And
what is your index size when things to wonky? (approximately)
I can say that other people create very large indexes without this
happening,
but the only thing that says is that this isn't a *known* problem.
Is there any chance you're re-opening the indexwriter without
closing the old one (not that you should, you should reuse the writer,
but I've put a new in the wrong place by mistake a time or two)....
But I confess that I have no idea what happens under the covers when
indexing,
so I'll have to leave any real insights to the folks who know.
Erick
On Fri, Apr 3, 2009 at 8:41 AM, John Byrne <john.by...@propylon.com> wrote:
The maximum JVM memory is 2GB. Apparently 1.2GB is being used up by this
class.
All IndexWriter settings are left as default.
I haven't tried any changes yet, because the problem so far has on ly
happened in a production environment that I can't mess with. I am planning
to try reproducing it locally soon, but it takes quite a while before it
happens.
-John
Erick Erickson wrote:
How much memory are you allocating for the JVM? And what are your
various indexwriter settings (e.g. MaxBufferedDocs, MaxMergeDocs, etc).
Have you tried different settings in setRamBufferSizeMB?
Best
Erick
On Fri, Apr 3, 2009 at 7:13 AM, John Byrne <john.by...@propylon.com>
wrote:
Hi, I'm having a problem where the JVM runs out of memory while indexing
a
large number of files. An analysis of the heapdump shows that most of the
memory was taken up with
"org/apache/lucene/util/ScorerDocQueue$HeapedScorerDoc".
I can't find any leaks in my code so far, and I was wondering, are there
any known problems with this class that could leak memory?
I'm using version 2.3.1.
Thanks!
-John
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org
------------------------------------------------------------------------
No virus found in this incoming message.
Checked by AVG - www.avg.com Version: 8.0.238 / Virus Database:
270.11.39/2038 - Release Date: 04/02/09 19:07:00
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org
------------------------------------------------------------------------
No virus found in this incoming message.
Checked by AVG - www.avg.com
Version: 8.0.238 / Virus Database: 270.11.40/2039 - Release Date: 04/03/09 06:19:00
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org
