AW: OutOfMemoryError using IndexWriter

2009-06-25 Thread stefan
etreff: Re: OutOfMemoryError using IndexWriter Interesting that excessive deletes buffering is not your problem... Even if you can't post the resulting test case, if you can simplify it & run locally, to rule out anything outside Lucene that's allocating the byte/char/byte[] arrays, that ca

Re: OutOfMemoryError using IndexWriter

2009-06-25 Thread Michael McCandless
Michael McCandless [mailto:luc...@mikemccandless.com] > Gesendet: Do 25.06.2009 13:13 > An: java-user@lucene.apache.org > Betreff: Re: OutOfMemoryError using IndexWriter > > Can you post your test code?  If you can make it a standalone test, > then I can repro and dig down faster.

AW: OutOfMemoryError using IndexWriter

2009-06-25 Thread stefan
it is similar to creating a new IndexWriter. HTH, Stefan -Ursprüngliche Nachricht- Von: Michael McCandless [mailto:luc...@mikemccandless.com] Gesendet: Do 25.06.2009 13:13 An: java-user@lucene.apache.org Betreff: Re: OutOfMemoryError using IndexWriter Can you post your test code? If yo

Re: OutOfMemoryError using IndexWriter

2009-06-25 Thread Michael McCandless
OK it looks like no merging was done. I think the next step is to call IndexWriter.setMaxBufferedDeleteTerms(1000) and see if that prevents the OOM. Mike On Thu, Jun 25, 2009 at 7:16 AM, stefan wrote: > Hi, > > Here are the result of CheckIndex. I ran this just after I got the OOError. > > OK [4

AW: OutOfMemoryError using IndexWriter

2009-06-25 Thread stefan
Hi, Here are the result of CheckIndex. I ran this just after I got the OOError. OK [4 fields] test: terms, freq, prox...OK [509534 terms; 9126904 terms/docs pairs; 4933036 tokens] test: stored fields...OK [148124 total field count; avg 2 fields per doc] test: term vectors...

Re: OutOfMemoryError using IndexWriter

2009-06-25 Thread Simon Willnauer
rays, I will need some >> more time for this. >> >> Stefan >> >> -Ursprüngliche Nachricht- >> Von: Michael McCandless [mailto:luc...@mikemccandless.com] >> Gesendet: Mi 24.06.2009 17:50 >> An: java-user@lucene.apache.org >> Betreff: Re: OutO

Re: OutOfMemoryError using IndexWriter

2009-06-25 Thread Michael McCandless
time for this. > > Stefan > > -Ursprüngliche Nachricht- > Von: Michael McCandless [mailto:luc...@mikemccandless.com] > Gesendet: Mi 24.06.2009 17:50 > An: java-user@lucene.apache.org > Betreff: Re: OutOfMemoryError using IndexWriter > > On Wed, Jun 24, 2009 at 10:1

AW: OutOfMemoryError using IndexWriter

2009-06-25 Thread stefan
...@mikemccandless.com] Gesendet: Mi 24.06.2009 17:50 An: java-user@lucene.apache.org Betreff: Re: OutOfMemoryError using IndexWriter On Wed, Jun 24, 2009 at 10:18 AM, stefan wrote: > > Hi, > > >>OK so this means it's not a leak, and instead it's just that stuff

Re: OutOfMemoryError using IndexWriter

2009-06-25 Thread Michael McCandless
On Thu, Jun 25, 2009 at 3:02 AM, stefan wrote: >>But a "leak" would keep leaking over time, right?  Ie even a 1 GB heap >>on your test db should eventually throw OOME if there's really a leak. > No not necessarily, since I stop indexing ones everything is indexed - I > shall try repeated runs wit

AW: OutOfMemoryError using IndexWriter

2009-06-25 Thread stefan
Hi, >But a "leak" would keep leaking over time, right? Ie even a 1 GB heap >on your test db should eventually throw OOME if there's really a leak. No not necessarily, since I stop indexing ones everything is indexed - I shall try repeated runs with 120MB. >Are you calling updateDocument (which

Re: OutOfMemoryError using IndexWriter

2009-06-24 Thread Michael McCandless
On Wed, Jun 24, 2009 at 10:23 AM, stefan wrote: > does Lucene keep the complete index in memory ? No. Certain things (deleted docs, norms, field cache, terms index) are loaded into memory, but these are tiny compared to what's not loaded into memory (postings, stored docs, term vectors). > As s

Re: OutOfMemoryError using IndexWriter

2009-06-24 Thread Michael McCandless
On Wed, Jun 24, 2009 at 10:18 AM, stefan wrote: > > Hi, > > >>OK so this means it's not a leak, and instead it's just that stuff is >>consuming more RAM than expected. > Or that my test db is smaller than the production db which is indeed the case. But a "leak" would keep leaking over time, right?

RE: OutOfMemoryError using IndexWriter

2009-06-24 Thread Sudarsan, Sithu D.
: stefan [mailto:ste...@intermediate.de] Sent: Wednesday, June 24, 2009 10:23 AM To: java-user@lucene.apache.org Subject: AW: OutOfMemoryError using IndexWriter Hi, does Lucene keep the complete index in memory ? As stated before the result index is 50MB, this would correlate with the memory

AW: OutOfMemoryError using IndexWriter

2009-06-24 Thread stefan
to open. Please post your results/views. Sincerely, Sithu -Original Message- From: stefan [mailto:ste...@intermediate.de] Sent: Wednesday, June 24, 2009 10:08 AM To: java-user@lucene.apache.org Subject: AW: OutOfMemoryError using IndexWriter Hi, I do use Win32. What do you mean by

RE: OutOfMemoryError using IndexWriter

2009-06-24 Thread Sudarsan, Sithu D.
open. Please post your results/views. Sincerely, Sithu -Original Message- From: stefan [mailto:ste...@intermediate.de] Sent: Wednesday, June 24, 2009 10:08 AM To: java-user@lucene.apache.org Subject: AW: OutOfMemoryError using IndexWriter Hi, I do use Win32. What do you mean by

AW: OutOfMemoryError using IndexWriter

2009-06-24 Thread stefan
Hi, >OK so this means it's not a leak, and instead it's just that stuff is >consuming more RAM than expected. Or that my test db is smaller than the production db which is indeed the case. >Hmm -- there are quite a few buffered deletes pending. It could be we >are under-accounting for RAM used

AW: OutOfMemoryError using IndexWriter

2009-06-24 Thread stefan
sendet: Mi 24.06.2009 15:55 An: java-user@lucene.apache.org Betreff: RE: OutOfMemoryError using IndexWriter Hi Stefan, Are you using Windows 32 bit? If so, sometimes, if the index file before optimizations crosses your jvm memory usage settings (if say 512MB), there is a possibility of this

AW: OutOfMemoryError using IndexWriter

2009-06-24 Thread stefan
ne 24, 2009 4:08:43 AM > Subject: OutOfMemoryError using IndexWriter > > Hi, > > I am using Lucene 2.4.1 to index a database with less than a million records. > The resulting index is about 50MB in size. > I keep getting an OutOfMemory Error if I re-use the same IndexWriter to

RE: OutOfMemoryError using IndexWriter

2009-06-24 Thread Sudarsan, Sithu D.
-2587 sithu.sudar...@fda.hhs.gov sdsudar...@ualr.edu -Original Message- From: stefan [mailto:ste...@intermediate.de] Sent: Wednesday, June 24, 2009 4:09 AM To: java-user@lucene.apache.org Subject: OutOfMemoryError using IndexWriter Hi, I am using Lucene 2.4.1 to index a database with less

Re: OutOfMemoryError using IndexWriter

2009-06-24 Thread Michael McCandless
On Wed, Jun 24, 2009 at 7:43 AM, stefan wrote: > I tried with 100MB heap size and got the Error as well, it runs fine with > 120MB. OK so this means it's not a leak, and instead it's just that stuff is consuming more RAM than expected. > Here is the histogram (application classes marked with --

Re: OutOfMemoryError using IndexWriter

2009-06-24 Thread Otis Gospodnetic
24, 2009 4:08:43 AM > Subject: OutOfMemoryError using IndexWriter > > Hi, > > I am using Lucene 2.4.1 to index a database with less than a million records. > The resulting index is about 50MB in size. > I keep getting an OutOfMemory Error if I re-use the same IndexWriter to

AW: OutOfMemoryError using IndexWriter

2009-06-24 Thread stefan
tefan -Ursprüngliche Nachricht- Von: Michael McCandless [mailto:luc...@mikemccandless.com] Gesendet: Mi 24.06.2009 11:52 An: java-user@lucene.apache.org Betreff: Re: OutOfMemoryError using IndexWriter Hmm -- I think your test env (80 MB heap, 50 MB used by app + 16 MB IndexWriter RAM buffe

Re: OutOfMemoryError using IndexWriter

2009-06-24 Thread Michael McCandless
)      3268608 (size) > > Well, something I should do differently ? > > Stefan > > -Ursprüngliche Nachricht- > Von: Michael McCandless [mailto:luc...@mikemccandless.com] > Gesendet: Mi 24.06.2009 10:48 > An: java-user@lucene.apache.org > Betreff: Re: OutOfMemory

AW: OutOfMemoryError using IndexWriter

2009-06-24 Thread stefan
: OutOfMemoryError using IndexWriter How large is the RAM buffer that you're giving IndexWriter? How large a heap size do you give to JVM? Can you post one of the OOM exceptions you're hitting? Mike On Wed, Jun 24, 2009 at 4:08 AM, stefan wrote: > Hi, > > I am using Lucene 2.4.1 to in

Re: OutOfMemoryError using IndexWriter

2009-06-24 Thread Michael McCandless
How large is the RAM buffer that you're giving IndexWriter? How large a heap size do you give to JVM? Can you post one of the OOM exceptions you're hitting? Mike On Wed, Jun 24, 2009 at 4:08 AM, stefan wrote: > Hi, > > I am using Lucene 2.4.1 to index a database with less than a million records

OutOfMemoryError using IndexWriter

2009-06-24 Thread stefan
Hi, I am using Lucene 2.4.1 to index a database with less than a million records. The resulting index is about 50MB in size. I keep getting an OutOfMemory Error if I re-use the same IndexWriter to index the complete database. This is though recommended in the performance hints. What I now do is