at 8:27 AM, ajay_gupta
>>> wrote:
>>> >>>>
>>> >>>>>
>>> >>>>> Hi,
>>> >>>>> It might be general question though but I couldn't find the answer
>>> yet.
>>> >>>>> I
&g
ve around 90k documents sizing around 350 MB. Each document
>> contains
>> >>>>> a
>> >>>>> record which has some text content. For each word in this text I
>> want
>> >>>>> to
>> >>>>> store context for that word a
n of this method and I
>>>>> observed that after each call of update_context memory increases and
>>>>> when
>>>>> it
>>>>> reaches around 65-70k it goes outofmemory so somewhere memory is
>>>>> increasing
>>>>>
ystem.gc() to release
>>>> memory
>>>> and I also tried various other parameters like
>>>> context_writer.setMaxBufferedDocs()
>>>> context_writer.setMaxMergeDocs()
>>>> context_writer.setRAMBufferSizeMB()
>>>> I set these parame
rameters smaller values as well but nothing worked.
>>>
>>> Any hint will be very helpful.
>>>
>>> Thanks
>>> Ajay
>>>
>>>
>>> Michael McCandless-2 wrote:
>>> >
>>> > The worst case RAM usage for Lucene i
ument -- it must flush after the doc has been fully
>> > indexed.
>> >
>> > This past thread (also from Paul) delves into some of the details:
>> >
>> > http://lucene.markmail.org/thread/pbeidtepentm6mdn
>> >
>> > But it's not clear whe
gt; > more details about the docs, or, some code fragments, could help shed
> > light.
> >
> > Mike
> >
> > On Tue, Mar 2, 2010 at 8:47 AM, Murdoch, Paul
> > wrote:
> >> Ajay,
> >>
> >> Here is another thread I started on the same issue.
> >
e.
>>
>> http://stackoverflow.com/questions/1362460/why-does-lucene-cause-oom-whe
>> n-indexing-large-files
>>
>> Paul
>>
>>
>> -Original Message-
>> From: java-user-return-45254-paul.b.murdoch=saic....@lucene.apache.org
>> [mailto
5254-paul.b.murdoch=saic@lucene.apache.org
> [mailto:java-user-return-45254-paul.b.murdoch=saic@lucene.apache.org
> ] On Behalf Of ajay_gupta
> Sent: Tuesday, March 02, 2010 8:28 AM
> To: java-user@lucene.apache.org
> Subject: Lucene Indexing out of memory
>
>
> Hi,
&g
> and
> >>>>> for each word in that document I am appending fixed number of
> >>>>> surrounding
> >>>>> words. To do that first I search in existing indices if this word
> >>>>> already
> >
d the new context
>>>>> and
>>>>> update the document. In case no context exist I create a document with
>>>>> fields "word" and "context" and add these two fields with values as
>>>>> word
>>>>> value and
t;> fields "word" and "context" and add these two fields with values as
>>>> word
>>>> value and context value.
>>>>
>>>> I tried this in RAM but after certain no of docs it gave out of memory
>>>>
;> error
> >> so I thought to use FSDirectory method but surprisingly after 70k
> >> documents
> >> it also gave OOM error. I have enough disk space but still I am getting
> >> this
> >> error.I am not sure even for disk b
; documents
>>> it also gave OOM error. I have enough disk space but still I am getting
>>> this
>>> error.I am not sure even for disk based indexing why its giving this
>>> error.
>>> I thought disk based indexing will be slow but atleast it will be
>>
disk space but still I am getting
>> this
>> error.I am not sure even for disk based indexing why its giving this
>> error.
>> I thought disk based indexing will be slow but atleast it will be
>> scalable.
>> Could so
-paul.b.murdoch=saic@lucene.apache.org
] On Behalf Of ajay_gupta
Sent: Tuesday, March 02, 2010 8:28 AM
To: java-user@lucene.apache.org
Subject: Lucene Indexing out of memory
Hi,
It might be general question though but I couldn't find the answer yet.
I
have around 90k documents sizing around 3
cene.apache.org
] On Behalf Of ajay_gupta
Sent: Tuesday, March 02, 2010 8:28 AM
To: java-user@lucene.apache.org
Subject: Lucene Indexing out of memory
Hi,
It might be general question though but I couldn't find the answer yet.
I
have around 90k documents sizing around 350 MB. Each document con
e the issue ?
>
> Thanks
> Ajay
> --
> View this message in context:
> http://old.nabble.com/Lucene-Indexing-out-of-memory-tp27755872p27755872.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
>
p://old.nabble.com/Lucene-Indexing-out-of-memory-tp27755872p27755872.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
-
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional comm
19 matches
Mail list logo
