Your stack trace is incomplete: it doesn't even show where the OOM occurred.
On Sun, Apr 14, 2013 at 7:48 PM, Wei Wang <welshw...@gmail.com> wrote: > Unfortunately, I got another problem. My index has 9 segments (9 dvdd > files) with total size is about 22GB. The merging step eventually failed > and I saw an error message: > > Exception in thread "main" java.lang.IllegalStateException: this writer hit > an OutOfMemoryError; cannot complete forceMerge > at > org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1664) > at > org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1610) > at > > com.ea.eadp.data.aem.audience.indexer.tools.IndexingTool.mergeIndex(IndexingTool.java:196) > at > > com.ea.eadp.data.aem.audience.indexer.tools.AudienceIndexer.main(AudienceIndexer.java:46) > Exception in thread "Lucene Merge Thread #0" > org.apache.lucene.index.MergePolicy$MergeException: > java.lang.OutOfMemoryError: Java heap space > at > > org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:541) > at > > org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:514) > > I configured jvm with "-Xmx4096m", and it seems still not enough memory. I > thought DiskDocValuesFormat puts most of the data on disk and there should > not be that much memory consumption. But it seems not the case. > > On Sun, Apr 14, 2013 at 4:13 PM, Wei Wang <welshw...@gmail.com> wrote: > > > That makes sense. > > > > BTW, I checked the jar file. Exactly as you pointed out, the services > > files only contains info from lucene-core, without codec from > > lucene-codecs. After adding the maven plugin, now it is running. > > > > Thanks! > > > > > > On Sun, Apr 14, 2013 at 3:26 PM, Uwe Schindler <u...@thetaphi.de> wrote: > > > >> Hi, > >> > >> > Thanks for the hint. I will double check the jar file. > >> > > >> > I am just a bit puzzled that if the indexing step recognizes 'Disk' > >> codec and > >> > creates index properly, the merge step that immediately follows > indexing > >> > seems should also recognize the 'Disk' codec. > >> > >> This is easy to explain: By creating the custom Lucene42 Codec as a > >> Class, you just define the disk format on the initial write (when *new* > >> segments are written with new documents). While merging (or > force-merging), > >> Lucene uses the metadata that’s already on disk for the segments to > merge. > >> The metadata on disk contains the names of all codec components used. > Those > >> metadata is also used when opening IndexReaders. It will then use SPI > and > >> META-INF/services files to look up the class that is responsible for > e.g. > >> the "Disk" docvalues format. Without the META-INF data, Lucene cannot > >> lookup the segment codecs. > >> > >> Uwe > >> > >> > On Sun, Apr 14, 2013 at 3:03 PM, Uwe Schindler <u...@thetaphi.de> > wrote: > >> > > >> > > Are you sure that you use the ServicesResourceTransformer in your > >> > > shade config? > >> > > > >> > > > >> > > http://maven.apache.org/plugins/maven-shade- > >> > plugin/examples/resource-t > >> > > ransformers.html#ServicesResourceTransformer > >> > > > >> > > The problem is: lucene-core.jar and lucene-codecs.jar both contain > >> > > codec components and their classes are listed in META-INF/services. > If > >> > > those files are not correctly merged through this resource > >> > > transformer, the resulting JAR file will miss some codecs. > >> > > > >> > > You can check correctness by opening the final JAR file with a ZIP > >> > > program and check that all files in META-INF/services contain all > >> > > entries merged from all Lucene JARs. > >> > > > >> > > Uwe > >> > > > >> > > ----- > >> > > Uwe Schindler > >> > > H.-H.-Meier-Allee 63, D-28213 Bremen > >> > > http://www.thetaphi.de > >> > > eMail: u...@thetaphi.de > >> > > > >> > > > >> > > > -----Original Message----- > >> > > > From: Wei Wang [mailto:welshw...@gmail.com] > >> > > > Sent: Sunday, April 14, 2013 11:49 PM > >> > > > To: java-user@lucene.apache.org > >> > > > Subject: Re: DiskDocValuesFormat > >> > > > > >> > > > Yes, I used Maven Shade plugin, but still have this problem. Here > is > >> > > > the Maven output during packaging: > >> > > > > >> > > > [INFO] --- maven-shade-plugin:2.0:shade (default) @ > >> > > > audience-profile- indexer --- [INFO] Including > >> > > > commons-collections:commons- > >> > > > collections:jar:3.2.1 in the shaded jar. > >> > > > [INFO] Including org.mockito:mockito-core:jar:1.9.5 in the shaded > >> jar. > >> > > > [INFO] Including org.hamcrest:hamcrest-core:jar:1.1 in the shaded > >> jar. > >> > > > [INFO] Including org.objenesis:objenesis:jar:1.0 in the shaded > jar. > >> > > > [INFO] Including junit:junit:jar:4.11 in the shaded jar. > >> > > > [INFO] Including log4j:log4j:jar:1.2.17 in the shaded jar. > >> > > > [INFO] Including org.apache.lucene:lucene-core:jar:4.2.1 in the > >> > > > shaded > >> > > jar. > >> > > > [INFO] Including org.apache.lucene:lucene-queries:jar:4.2.1 in the > >> > > > shaded jar. > >> > > > [INFO] Including org.apache.lucene:lucene-queryparser:jar:4.2.1 in > >> > > > the shaded jar. > >> > > > [INFO] Including org.apache.lucene:lucene-sandbox:jar:4.2.1 in the > >> > > > shaded jar. > >> > > > [INFO] Including jakarta-regexp:jakarta-regexp:jar:1.4 in the > >> shaded jar. > >> > > > [INFO] Including > org.apache.lucene:lucene-analyzers-common:jar:4.2.1 > >> > > > in the shaded jar. > >> > > > [INFO] Including org.apache.lucene:lucene-codecs:jar:4.2.1 in the > >> > > > shaded > >> > > jar. > >> > > > [INFO] Including commons-lang:commons-lang:jar:2.6 in the shaded > >> jar. > >> > > > [INFO] Including commons-logging:commons-logging:jar:1.1.1 in the > >> > > > shaded jar. > >> > > > [INFO] Including commons-io:commons-io:jar:2.4 in the shaded jar. > >> > > > [INFO] Replacing original artifact with shaded artifact. > >> > > > > >> > > > On Sun, Apr 14, 2013 at 2:40 PM, Uwe Schindler <u...@thetaphi.de> > >> > wrote: > >> > > > > >> > > > > If you create a single JAR file out of multiple Lucene JAR files > >> > > > > use a tool like Maven Shade plugin, otherwise, required metadata > >> > > > > propreties > >> > > > > (META-INF/services) files in the JAR files are not correctly > >> > > > > merged together. > >> > > > > > >> > > > > ----- > >> > > > > Uwe Schindler > >> > > > > H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de > >> > > > > eMail: u...@thetaphi.de > >> > > > > > >> > > > > > >> > > > > > -----Original Message----- > >> > > > > > From: Wei Wang [mailto:welshw...@gmail.com] > >> > > > > > Sent: Sunday, April 14, 2013 11:30 PM > >> > > > > > To: java-user@lucene.apache.org > >> > > > > > Subject: Re: DiskDocValuesFormat > >> > > > > > > >> > > > > > Hi Adrien, > >> > > > > > > >> > > > > > The Lucene42Codec works well to generate the index with > >> > > > > > DiskDocValuesFormat. But when I tried to merge the index > >> segments > >> > by > >> > > > > > calling: > >> > > > > > > >> > > > > > IndexWriter iw = new IndexWriter(directory, iw_config); ... > >> > > > > > iw.forceMerge(1); > >> > > > > > > >> > > > > > I got the following error message: > >> > > > > > > >> > > > > > Caused by: java.lang.IllegalArgumentException: A SPI class of > >> type > >> > > > > > org.apache.lucene.codecs.DocValuesFormat with name 'Disk' does > >> > not > >> > > > exist. > >> > > > > > You need to add the corresponding JAR file supporting this SPI > >> to > >> > > > > > your classpath.The current classpath supports the following > >> names: > >> > > > > > [Lucene42] > >> > > > > > > >> > > > > > Any hint on this classpath problem? I have created a single > jar > >> file > >> > > > > that has all > >> > > > > > necessary dependencies, such as lucene-codecs-4.2.0.jar. And I > >> > > > > > assume the indexing step works well, so Lucene already knows > the > >> > > > > > format with name 'Disk'. > >> > > > > > > >> > > > > > Thanks. > >> > > > > > > >> > > > > > On Sat, Apr 13, 2013 at 4:25 AM, Adrien Grand < > >> jpou...@gmail.com> > >> > > > wrote: > >> > > > > > > >> > > > > > > Hi Wei, > >> > > > > > > > >> > > > > > > On Sat, Apr 13, 2013 at 7:44 AM, Wei Wang > >> > <welshw...@gmail.com> > >> > > > > > wrote: > >> > > > > > > > I am trying to use DiskDocValuesFormat for a particular > >> > > > > > > > BinaryDocValuesField. It seems there is no good examples > >> > showing > >> > > > > > > > how to > >> > > > > > > do > >> > > > > > > > this. The only hint I got from various docs and forums is > >> set > >> > > > > > > > some codec > >> > > > > > > in > >> > > > > > > > IndexWriter. Could someone give a few lines of code > snippet > >> and > >> > > > > > > > show how > >> > > > > > > to > >> > > > > > > > set DiskDocValuesFormat? > >> > > > > > > > >> > > > > > > Lucene42Codec can be extended to specify the doc values > format > >> > to > >> > > > > > > use on a per-field basis. For example: > >> > > > > > > > >> > > > > > > final Codec codec = new Lucene42Codec() { > >> > > > > > > final Lucene42DocValuesFormat memoryDVFormat = new > >> > > > > > > Lucene42DocValuesFormat(); > >> > > > > > > final DiskDocValuesFormat diskDVFormat = new > >> > > > DiskDocValuesFormat(); > >> > > > > > > @Override > >> > > > > > > public DocValuesFormat getDocValuesFormatForField(String > >> field) > >> > { > >> > > > > > > if ("dv_mem".equals(field)) { > >> > > > > > > // use Lucene42 for "dv_mem" > >> > > > > > > return memoryDVFormat; > >> > > > > > > } else { > >> > > > > > > // use Disk otherwise > >> > > > > > > return diskDVFormat; > >> > > > > > > } > >> > > > > > > } > >> > > > > > > }; > >> > > > > > > > >> > > > > > > Then just pass this Codec instance to your > IndexWriterConfig. > >> > > > > > > > >> > > > > > > -- > >> > > > > > > Adrien > >> > > > > > > > >> > > > > > > > >> ------------------------------------------------------------------ > >> > > > > > > --- To unsubscribe, e-mail: > >> > > > > > > java-user-unsubscr...@lucene.apache.org > >> > > > > > > For additional commands, e-mail: java-user- > >> > h...@lucene.apache.org > >> > > > > > > > >> > > > > > > > >> > > > > > >> > > > > > >> > > > > > >> --------------------------------------------------------------------- > >> > > > > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > >> > > > > For additional commands, e-mail: > java-user-h...@lucene.apache.org > >> > > > > > >> > > > > > >> > > > >> > > > >> > > > --------------------------------------------------------------------- > >> > > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > >> > > For additional commands, e-mail: java-user-h...@lucene.apache.org > >> > > > >> > > > >> > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > >> For additional commands, e-mail: java-user-h...@lucene.apache.org > >> > >> > > >
