Phew!  Thank for bringing closure Ross.  Happy indexing,

Mike

On Wed, May 19, 2010 at 12:50 PM, Woolf, Ross <ross_wo...@bmc.com> wrote:
> Just wanted to report that Michael was able to find the issue that was 
> plaguing us, He has checked fixes into the 2.9.x, 3.0.x, 3.1.x, 4.0.x 
> branches.  Most of the issues were related to indexing documents larger than 
> the indexing buffer size (16mb by default).  Now we no longer run out of 
> memory during our large document indexing runs.
>
> Thanks for your help Michael in resolving this.
>
> -----Original Message-----
> From: Michael McCandless [mailto:luc...@mikemccandless.com]
> Sent: Friday, May 14, 2010 11:23 AM
> To: java-user@lucene.apache.org
> Subject: Re: IndexWriter and memory usage
>
> The patch looks correct.
>
> The 16 MB RAM buffer means the sum of the shared char[], byte[] and
> PostingList/RawPostingList memory will be kept under 16 MB.  There are
> definitely other things that require memory beyond this -- eg during a
> segment merge, SegmentReaders are opened for each segment being
> merged.  Also, if there are pending deletions, 4 bytes per doc is
> allocated.
>
> Applying deletions also opens SegmentReaders.
>
> Also: a single very large document will cause IW to blow way past the
> 16 MB limit, using up as much as is required to index that one doc.
> When that doc is finished, it will then flush and free objects until
> it's back under the 16 MB limit.  If several threads happen to index
> large docs at the same time, the problem is that much worse (they all
> must finish before IW can flush).
>
> Can you print the size of the documents you're indexing and see if
> that correlates to when you see the memory growth?
>
> Mike
>
> On Tue, May 11, 2010 at 2:57 PM, Woolf, Ross <ross_wo...@bmc.com> wrote:
>> Still working on some of the things you ask, namely searching without 
>> indexing.  I need to modify our code and the general indexing process takes 
>> 2 hours, so I won't have a quick turn around on that.  We also have a hard 
>> time answering the question about items that are normal but do they use more 
>> than the 16 MB.  The heap dump does not allow us to quickly identify 
>> specifics on objects like we show in images below, so we really don't know 
>> what the amount of memory is used up in objects of this sort.  We only know 
>> that byte[] total for all is at 197891887.
>>
>> However, I have provided another image that breaks down the memory usage 
>> from the heap.   A big question we have is that we talk about the 16 mb 
>> buffer, but is there other memory used by Lucene beyond that that we should 
>> expect to see?
>>
>> http://i39.tinypic.com/o0o560.jpg
>>
>> we have 197891887 used in byte[] (anyone we look at is related in some way 
>> to the index writer)
>> we have 169263904 used in char[] (these are related to the index writer too)
>> we have 72658944 used in FreqProxTermsWriter$PostingList
>> we have 37722668 used in RawPostingList[]
>>
>> All of these are well over the 16mb.  So we are a little lost as to what we 
>> should expect to see when we look at the memory usage
>>
>> I've attached the patch and the CheckIndex files.  Unfortunately on the 
>> patch I guess my editor made some space line changes, so you get a lot of 
>> extra items in the patch that really are not any changes other than 
>> tab/space.
>>
>> If you are open to a live share again, then maybe you could look at this 
>> data quicker than the screen shots I send.
>>
>> -----Original Message-----
>> From: Michael McCandless [mailto:luc...@mikemccandless.com]
>> Sent: Monday, May 10, 2010 2:27 AM
>> To: java-user@lucene.apache.org
>> Subject: Re: IndexWriter and memory usage
>>
>> Hmmmm...
>>
>> Your usage (searching for old doc & updating it, to add new fields) is fine.
>>
>> But: what memory usage do you see if you open a searcher, and search
>> for all docs, but don't open an IndexWriter?  We need to tease apart
>> the IndexReader vs IndexWriter memory usage you are seeing.  Also, can
>> you post the output of CheckIndex (java
>> org.apache.lucene.index.CheckIndex /path/to/index) of your fully built
>> index?  That may give some hints about expected memory usage of IR (eg
>> if # unique terms is large).
>>
>> More comments below:
>>
>> On Thu, May 6, 2010 at 1:03 PM, Woolf, Ross <ross_wo...@bmc.com> wrote:
>>> Sorry to be so long in getting back on this. The patch you provided has 
>>> improved the situation but we are still seeing some memory loss.  The 
>>> following are some images from the heap dump.  I'll share with you what we 
>>> are seeing now.
>>>
>>> This first image shows the memory pattern.  Our fist commit takes place at 
>>> about 3:54 when the steady trend up takes a drop and the new cycle begins.  
>>>  What we have found is the 2422 fix has made the memory in the first phase 
>>> before the commit much better (and I'm sure throughout the entire run).  
>>> But as you can see after the commit then we then again begin to lose 
>>> memory.  One of the pieces of info to know about this is what you are 
>>> seeing, we have 5 threads that are pushing data to our Lucene plugin.  If 
>>> we drop it down to 1 thread then we are much more successful and can 
>>> actually index all of our data without running out of memory but at 5 
>>> threads it gets into trouble.  We still see a trend up in memory usage, but 
>>> not as severe as when we use the multiple threads.
>>> http://tinypic.com/view.php?pic=2w6bf68&s=5
>>
>> Can you post the output of "svn diff" on the 2.9 code base you're
>> using?  I just want to look & verify all issues we've discussed are
>> included in your changes.  The fact that 1 thread is fine and 5
>> threads are not still sounds like a symptom of LUCENE-2283.
>>
>> Also, does that heap usage graph exclude garbage?  Or, alternatively,
>> can you provoke an OOME w/ 512 MB heap and then capture the heap dump
>> at that point?
>>
>>> There is another piece of the picture that I think might be coming into 
>>> play.  We have plugged Lucene into a legacy app and are subject to how we 
>>> can get it to deliver the data that we are indexing.  In some scenarios 
>>> (like the one we are having this problem with) we are building our 
>>> documents progressively (adding fields to the document through the 
>>> process).  What you see before the first commit is the legacy system 
>>> handing us the first field for many documents. Once we have gotten all of 
>>> "field 1" for all documents then we commit that data into the index.  Then 
>>> the system starts feeding us "field 2."  So we perform a search to see if 
>>> the document already exists (for the scenario you are seeing it does) and 
>>> so it retrieves the original document (we store a document ID) and it then 
>>> adds the new field of data to the existing document and we "update" the 
>>> document in the index.  After the first commit, the rest of the process is 
>>> one where a document already exist and so the new field is added and and 
>>> the document is updated.  It is in this process that we start rapidly 
>>> losing memory.  The following images show some examples of common areas 
>>> where memory is being held.
>>>
>>> http://tinypic.com/view.php?pic=11vkwnb&s=5
>>
>> This looks like "normal" memory usage of IndexWriter -- these are the
>> recycled buffers used for holding stored fields.  However: the net RAM
>> used by this allocation should not exceed your 16 MB IW ram buffer
>> size -- does it?
>>
>>> http://tinypic.com/view.php?pic=abq9fp&s=5
>>
>> This one is the byte[] buffer used by CompoundFileReader, opened by
>> IndexReader.  It's odd that you have so many of these (if I'm reading
>> this correctly) -- are you certain all opened readers are being
>> closed?  How many segments do you have in your index?  Or... are there
>> many unique threads doing the searching?  EG do you create a new
>> thread for every search or update?
>>
>>> http://tinypic.com/view.php?pic=25pskyp&s=5
>>
>> This one is also normal memory used by IndexWriter, but as above, the
>> net RAM used by this allocation (summed w/ the above one) should not
>> exceed your 16 MB IW ram buffer size.
>>
>>> As mentioned, we are subject to how we can have the legacy app feed us the 
>>> data and so this is why we do it this way.  We treat this system as a real 
>>> time system and at anytime the legacy system may send us a field that needs 
>>> to be added or updated to a document.  So we search for the document and if 
>>> found we either add or update a field if the field is already existing in 
>>> the document.  So I started to wonder if a clue in this memory loss comes 
>>> from the fact that we are retrieving an existing document and then adding 
>>> to it and updating.
>>>
>>> Now, if we eliminate the updating and simply add each item as a new 
>>> document (which we did just to test but won't be adequate for our running 
>>> system), then we still see a slight trend upward in memory usage and the 
>>> following images show that now most of the  memory is consumed in char[] 
>>> rather than the byte[] we saw before.  We don't know if this is normal and 
>>> expected, or if it is something to be concerned about as well.
>>>
>>> http://tinypic.com/view.php?pic=vfgkyt&s=5
>>
>> That memory usage is normal -- it's used by the in-RAM terms index of
>> your opened IndexReader.  But I'd like to see the memory usage of
>> simply opening your IndexReader and searching for documents to update,
>> but not opening an IndexWriter at all.
>>
>> Mike
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: java-user-h...@lucene.apache.org
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: java-user-h...@lucene.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
> For additional commands, e-mail: java-user-h...@lucene.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
> For additional commands, e-mail: java-user-h...@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org

Reply via email to