Hi Mike, I ran a burn in test overnight with repeatedly indexing the same db in a loop. I set the heap size to 120MB and called setMaxBufferedDeleteTerms( 1000), I did not call commit and used the same index writer. This test passed without any errors.
So to wrap this up - I shall call commit every 10000 documents added, the performance impact on calling setMaxBufferedDeleteTerms is quite significant (I guess somewhere between factor 2-10). I'll check on the speed of commit. Commit seems to be able to free more memory (since it runs with 100MB heap size). I'll have to give up on tracking down those binary references, I spent too much time on this already. Thanks a lot for your insights. Stefan -----Ursprüngliche Nachricht----- Von: Michael McCandless [mailto:luc...@mikemccandless.com] Gesendet: Do 25.06.2009 15:57 An: java-user@lucene.apache.org Betreff: Re: OutOfMemoryError using IndexWriter Interesting that excessive deletes buffering is not your problem... Even if you can't post the resulting test case, if you can simplify it & run locally, to rule out anything outside Lucene that's allocating the byte/char/byte[] arrays, that can help to isolate. Also, profilers can trace where allocations happened, eg YourKit. Mike On Thu, Jun 25, 2009 at 9:08 AM, stefan<ste...@intermediate.de> wrote: > Hi, > > I'm afraid my test setup and code this is far too big. > What I use lucene for is fairly simple. I have a database with about 150 > tables, I iterate all tables and create for each row a String representation > similar to a toString method containing all database data. This string is > then fed together with the primary key to lucene. Full-text search of my db > is then possible. Each document in Lucene represents a row in the database. > > I tried calling setMaxBufferedDeleteTerms with 100MB heap size to no avail, > but calling commit every 10000 documents does help. I assume a commit is > similar to creating a new IndexWriter. > > HTH, > > Stefan > > > > > -----Ursprüngliche Nachricht----- > Von: Michael McCandless [mailto:luc...@mikemccandless.com] > Gesendet: Do 25.06.2009 13:13 > An: java-user@lucene.apache.org > Betreff: Re: OutOfMemoryError using IndexWriter > > Can you post your test code? If you can make it a standalone test, > then I can repro and dig down faster. > > Can you try calling IndexWriter.setMaxBufferedDeleteTerms (eg, 1000) > and see if that prevents the OOM? > > Mike > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
--------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
