Michael McCandless [mailto:luc...@mikemccandless.com]
> Sent: Friday, May 14, 2010 11:23 AM
> To: java-user@lucene.apache.org
> Subject: Re: IndexWriter and memory usage
>
> The patch looks correct.
>
> The 16 MB RAM buffer means the sum of the shared char[], byte[] and
> Post
memory during our large
document indexing runs.
Thanks for your help Michael in resolving this.
-Original Message-
From: Michael McCandless [mailto:luc...@mikemccandless.com]
Sent: Friday, May 14, 2010 11:23 AM
To: java-user@lucene.apache.org
Subject: Re: IndexWriter and memory usage
The
ook at this data
> quicker than the screen shots I send.
>
> -Original Message-
> From: Michael McCandless [mailto:luc...@mikemccandless.com]
> Sent: Monday, May 10, 2010 2:27 AM
> To: java-user@lucene.apache.org
> Subject: Re: IndexWriter and memory usage
>
> H...
H...
Your usage (searching for old doc & updating it, to add new fields) is fine.
But: what memory usage do you see if you open a searcher, and search
for all docs, but don't open an IndexWriter? We need to tease apart
the IndexReader vs IndexWriter memory usage you are seeing. Also, can
yo
Sent: Thursday, April 29, 2010 2:07 PM
To: java-user@lucene.apache.org
Subject: Re: IndexWriter and memory usage
OK I think you may be hitting this:
https://issues.apache.org/jira/browse/LUCENE-2422
Since you have very large docs, the reuse that's done by
IndexInput/Output is tying u
ing this issue.
>
> -Original Message-
> From: Michael McCandless [mailto:luc...@mikemccandless.com]
> Sent: Tuesday, April 27, 2010 4:40 AM
> To: java-user@lucene.apache.org
> Subject: Re: IndexWriter and memory usage
>
> Oooh -- I suspect you are hitting this issue:
>
&
:28 PM, Woolf, Ross wrote:
>> How do I get to the 2.9.x branch? Every link I take from the Lucene site
>> takes me to the trunk which I assume is the 3.x version. I've tried to look
>> around svn but can't find anything labeled 2.9.x. Is there a daily build of
&
s things down, so we don't want to run like this,
but we wanted to test the behavior if we did so).
Thanks,
Ross
-Original Message-
From: Michael McCandless [mailto:luc...@mikemccandless.com]
Sent: Wednesday, April 14, 2010 2:52 PM
To: java-user@lucene.apache.org
Subject:
the fix you
> put into it, but I'm not sure where I get it from.
>
> -Original Message-
> From: Michael McCandless [mailto:luc...@mikemccandless.com]
> Sent: Wednesday, April 14, 2010 4:12 AM
> To: java-user@lucene.apache.org
> Subject: Re: IndexWriter and memory
ry out the fix you put into it,
but I'm not sure where I get it from.
-Original Message-
From: Michael McCandless [mailto:luc...@mikemccandless.com]
Sent: Wednesday, April 14, 2010 4:12 AM
To: java-user@lucene.apache.org
Subject: Re: IndexWriter and memory usage
It looks like t
sage-
> From: Woolf, Ross [mailto:ross_wo...@bmc.com]
> Sent: Tuesday, April 13, 2010 1:29 PM
> To: java-user@lucene.apache.org
> Subject: RE: IndexWriter and memory usage
>
> Are these fixes in 2.9x branch? We are using 2.9x and can't move to 3x just
> yet. If so,
apache.org
Subject: RE: IndexWriter and memory usage
Are these fixes in 2.9x branch? We are using 2.9x and can't move to 3x just
yet. If so, where do I specifically pick this up from?
-Original Message-
From: Lance Norskog [mailto:goks...@gmail.com]
Sent: Monday, April 12, 2010 10
t: Re: IndexWriter and memory usage
There is some bugs where the writer data structures retain data after
it is flushed. They are committed as of maybe the past week. If you
can pull the trunk and try it with your use case, that would be great.
On Mon, Apr 12, 2010 at 8:54 AM, Woolf, Ross wrote:
>
attachment). I'll see what I can do about reducing the
>> heap dump (It was supplied by a colleague).
>>
>>
>> -Original Message-
>> From: Michael McCandless [mailto:luc...@mikemccandless.com]
>> Sent: Saturday, April 03, 2010 3:39 AM
>> To
supplied by a colleague).
>
>
> -Original Message-
> From: Michael McCandless [mailto:luc...@mikemccandless.com]
> Sent: Saturday, April 03, 2010 3:39 AM
> To: java-user@lucene.apache.org
> Subject: Re: IndexWriter and memory usage
>
> Hmm why is the heap dump so immense?
urday, April 03, 2010 3:39 AM
> To: java-user@lucene.apache.org
> Subject: Re: IndexWriter and memory usage
>
> Hmm why is the heap dump so immense? Normally it contains the top N
> (eg 100) object types and their count/aggregate RAM usage.
>
> Can you attach the infoStream
To: java-user@lucene.apache.org
> Subject: Re: IndexWriter and memory usage
>
> Hmm, not good. Can you post a heap dump? Also, can you turn on
> infoStream, index up to the OOM @ 512 MB, and post the output?
>
> IndexWriter should not hang onto much beyond the RAM buffer.
5:21 PM
To: java-user@lucene.apache.org
Subject: Re: IndexWriter and memory usage
Hmm, not good. Can you post a heap dump? Also, can you turn on
infoStream, index up to the OOM @ 512 MB, and post the output?
IndexWriter should not hang onto much beyond the RAM buffer. But, it
does allocate and
Hmm, not good. Can you post a heap dump? Also, can you turn on
infoStream, index up to the OOM @ 512 MB, and post the output?
IndexWriter should not hang onto much beyond the RAM buffer. But, it
does allocate and then recycle this RAM buffer, so even in an idle
state (having indexed enough docs
We are seeing a situation where the IndexWriter is using up the Java Heap space
and only releases memory for garbage collection upon a commit. We are using
the default RAMBufferSize of 16 mb. We are using Lucene 2.9.1. We are set at
heap size of 512 mb.
We have a large number of documents th
20 matches
Mail list logo
