Thanks for showing interest Mike. The OOME comes in the middle of setting a value of one of the field in the doc. That field has a fairly large value. May be that could have been the reason.?
On Fri, Oct 3, 2008 at 4:57 PM, Michael McCandless < [EMAIL PROTECTED]> wrote: > > Note that large stored fields do not use up any RAM in IndexWriter's RAM > buffer because these stored fields are immediately written to the directory > and not stored in RAM for very long. > > Aditi, I'd love to see the full stack trace of the OOME that was originally > hit if you still have it... > > Mike > > > Ganesh wrote: > > Single document of 16 MB seems to be big. I think you are trying to store >> the entire document content. If it is so drop the stored field and store its >> reference information in the database, which could help to retreive the >> content later. >> >> Regards >> Ganesh >> >> ----- Original Message ----- From: "Aditi Goyal" <[EMAIL PROTECTED]> >> To: <java-user@lucene.apache.org> >> Sent: Friday, October 03, 2008 3:03 PM >> Subject: Re: Document larger than setRAMBufferSizeMB() >> >> >> Thanks Anshum. >>> Although it raises another query, committing the current buffer will >>> commit >>> the docs before and what will happen to the current doc which threw an >>> error >>> while adding a field to it, will that also get committed in the half?? >>> >>> Thanks a lot >>> Aditi >>> >>> On Fri, Oct 3, 2008 at 2:12 PM, Anshum <[EMAIL PROTECTED]> wrote: >>> >>> Hi Aditi, >>>> >>>> I guess increasing the buffer size would be a solution here, but in case >>>> you >>>> wouldn't know the expected max doc size. I guess the best way to handle >>>> that >>>> would be a regular try catch block in which you could commit the current >>>> buffer. At the least you could just continue the loop after doing >>>> whatever >>>> you wish to do using an exception handling block. >>>> >>>> -- >>>> Anshum Gupta >>>> Naukri Labs! >>>> http://ai-cafe.blogspot.com >>>> >>>> The facts expressed here belong to everybody, the opinions to me. The >>>> distinction is yours to draw............ >>>> >>>> On Fri, Oct 3, 2008 at 1:56 PM, Aditi Goyal <[EMAIL PROTECTED]> >>>> wrote: >>>> >>>> > Hi Everyone, >>>> > >>>> > I have an index which I am opening at one time only. I keep adding the >>>> > documents to it until I reach a limit of 500. >>>> > After this, I close the index and open it again. (This is done in >>>> order >>>> to >>>> > save time taken by opening and closing the index) >>>> > Also, I have set setRAMBufferSizeMB to 16MB. >>>> > >>>> > If the document size itself is greater than 16MB what will happen in > >>>> this >>>> > case?? >>>> > It is throwing >>>> > java.lang.OutOfMemoryError: Java heap space >>>> > Now, my query is, >>>> > Can we change something in the way we parse/index to make it more > >>>> memory >>>> > friendly so that it doesnt throw this exception. >>>> > And, Can it be caught and overcome gracefully? >>>> > >>>> > >>>> > Thanks a lot >>>> > Aditi >>>> > >>>> >>>> >> Send instant messages to your online friends >> http://in.messenger.yahoo.com >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [EMAIL PROTECTED] >> For additional commands, e-mail: [EMAIL PROTECTED] >> >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > >
