Ok, one customer of ours insists on running a Really Big Single Server Lucene index.
To be exact, there are about 300 million documents. This is running on a 64 bit JVM/64 bit OS with 24 GB(!) RAM allocated. Until very recently all was well. We then updated then in program version, which updated lucene 2.93 to lucene 3.02 Now, their searches are working fine IF you do not SORT the results. If you do SORT, you get stuff like 2011-08-30 13:01:31,489 [TP-Processor8] ERROR com.gwava.utils.ServerErrorHandlerStrategy - reportError: nastybadthing :: com.gwava.indexing.lucene.internal.LuceneSearchController.performSearchOperation:229 :: EXCEPTION : java.lang.OutOfMemoryError: Requested array size exceeds VM limit java.lang.OutOfMemoryError: Requested array size exceeds VM limit at org.apache.lucene.search.FieldCacheImpl$StringIndexCache.createValue(FieldCacheImpl.java:624) at org.apache.lucene.search.FieldCacheImpl$Cache.get(FieldCacheImpl.java:183) at org.apache.lucene.search.FieldCacheImpl.getStringIndex(FieldCacheImpl.java:611) at org.apache.lucene.search.FieldComparator$StringOrdValComparator.setNextReader(FieldComparator.java:805) at org.apache.lucene.search.TopFieldCollector$OneComparatorNonScoringCollector.setNextReader(TopFieldCollector.java:96) at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:209) at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:199) at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:177) at org.apache.lucene.search.Searcher.search(Searcher.java:49) at com.gwava.indexing.lucene.internal.LuceneSearchController.performSearchOperation(LuceneSearchController.java:223) at Looking at the sort class, the api docs appear to say it would create an element of 1.2 billion items (4*300m). Is this correct? Is the issue going beyond signed int32 limits of an array ( 2 billion items) or is it really a memory issue? How best to diagnose? Thanks --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
