For Sure. Lucene's explain is really expensive and is not purposed for production use, but only for rare troubleshooting. As a mitigation measure you can scroll result set by small portions more efficient like Hoss recently explained at SearchHub. In such kind of problems, usually it's possible to create sort of specialized custom collectors doing something particular.
Have a god day! On Sat, May 17, 2014 at 3:01 AM, Tom Burton-West <[email protected]> wrote: > Hello all, > > > I'm trying to get relevance scoring information for each of 1,000 docs > returned for each of 250 queries. If I run the query (appended below) > without debugQuery=on, I have no problem with getting all the results > with under 4GB of memory use. If I add the parameter &debugQuery=on, > memory use goes up continuously and after about 20 queries (with 1,000 > results each), memory use reaches about 29.1 GB and the garbage collector > gives up: > > " org.apache.solr.common.SolrException; null:java.lang.RuntimeException: > java.lang.OutOfMemoryError: GC overhead limit exceeded" > > I've attached a jmap -histo, exgerpt below. > > Is this a known issue with debugQuery? > > Tom > ---- > query: > > > q=Abraham+Lincoln&fl=id,score&indent=on&wt=json&start=0&rows=1000&version=2.2& > debugQuery=on > > without debugQuery=on: > > > q=Abraham+Lincoln&fl=id,score&indent=on&wt=json&start=0&rows=1000&version=2.2 > > num #instances #bytes Class description > -------------------------------------------------------------------------- > 1: 585,559 10,292,067,456 byte[] > 2: 743,639 18,874,349,592 char[] > 3: 53,821 91,936,328 long[] > 4: 70,430 69,234,400 int[] > 5: 51,348 27,111,744 > org.apache.lucene.util.fst.FST$Arc[] > 6: 286,357 20,617,704 org.apache.lucene.util.fst.FST$Arc > 7: 715,364 17,168,736 java.lang.String > 8: 79,561 12,547,792 * ConstMethodKlass > 9: 18,909 11,404,696 short[] > 10: 345,854 11,067,328 java.util.HashMap$Entry > 11: 8,823 10,351,024 * ConstantPoolKlass > 12: 79,561 10,193,328 * MethodKlass > 13: 228,587 9,143,480 > org.apache.lucene.document.FieldType > 14: 228,584 9,143,360 org.apache.lucene.document.Field > 15: 368,423 8,842,152 org.apache.lucene.util.BytesRef > 16: 210,342 8,413,680 java.util.TreeMap$Entry > 17: 81,576 8,204,648 java.util.HashMap$Entry[] > 18: 107,921 7,770,312 org.apache.lucene.util.fst.FST$Arc > 19: 13,020 6,874,560 > org.apache.lucene.util.fst.FST$Arc[] > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > -- Sincerely yours Mikhail Khludnev Principal Engineer, Grid Dynamics <http://www.griddynamics.com> <[email protected]>
