Hi, mike Is there any config option which let IndexReader to omit all norms and not to load norms to JVM heap?
Thanks & Best Regards! ------------------ Original ------------------ From: "lubin";<308181...@qq.com>; Date: Sat, Sep 13, 2014 08:20 PM To: "java-user"<java-user@lucene.apache.org>; Subject: Re: OutOfMemoryError throwed by SimpleMergedSegmentWarmer Hi, Mike In our use case, we have thousands of index fields, different kind of document have different fields. Do you meant that norms field will consume large memory? Why? If we decide to disable norms, do we need to rebuild our index entirely? By the way, We have 8 million documents and our jvm heap is 5G. Thanks & Best Regards! ------------------ Original ------------------ From: "Michael McCandless";<luc...@mikemccandless.com>; Date: Sat, Sep 13, 2014 06:29 PM To: "Lucene Users"<java-user@lucene.apache.org>; Subject: Re: OutOfMemoryError throwed by SimpleMergedSegmentWarmer The warmer just tries to load norms/docValues/etc. for all fields that have them enabled ... so this is likely telling you an IndexReader would also hit OOME. You either need to reduce the number of fields you have indexed, or at least disable norms (takes 1 byte per doc per indexed field regardless of whether that doc had indexed that field), or increase HEAP to the JVM. Mike McCandless http://blog.mikemccandless.com On Sat, Sep 13, 2014 at 4:25 AM, 308181687 <308181...@qq.com> wrote: > Hi, all > we got an OutOfMemoryError throwed by SimpleMergedSegmentWarmer. We use > lucene 4.7， and access index file by NRTCachingDirectory/MMapDirectory. Could > any body give me a hand? Strack trace is as follows: > > > > > > > org.apache.lucene.index.MergePolicy$MergeException: > java.lang.OutOfMemoryError: Java heap space > at > org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:518) > > > Caused by: java.lang.OutOfMemoryError: Java heap space > at > org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.loadNumeric(Lucene42DocValuesProducer.java:228) > at > org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.getNumeric(Lucene42DocValuesProducer.java:188) > at > org.apache.lucene.index.SegmentCoreReaders.getNormValues(SegmentCoreReaders.java:166) > at > org.apache.lucene.index.SegmentReader.getNormValues(SegmentReader.java:519) > at > org.apache.lucene.index.SimpleMergedSegmentWarmer.warm(SimpleMergedSegmentWarmer.java:52) > at > org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4275) > at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3743) > at > org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:405) > at > org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:482) > > > > > Thanks & Best Regards! --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
