On Wed, Oct 9, 2013 at 7:18 PM, Michael McCandless < luc...@mikemccandless.com> wrote:
> On Wed, Oct 9, 2013 at 7:13 PM, Benson Margulies <ben...@basistech.com> > wrote: > > On Tue, Oct 8, 2013 at 5:50 PM, Michael McCandless < > > luc...@mikemccandless.com> wrote: > > > >> DirectPostingsFormat? > >> > >> It stores all terms + postings as simple java arrays, uncompressed. > >> > > > > This definitely speeded things up in my benchmark, but I'm greedy for > more. > > I just made a codec that returns it as the postings guy, is that the > whole > > recipe?. Does it make sense to extend it any further to any of the other > > codec pieces? > > Yes, that's all you should need to do (you should have seen RAM usage > go up too, to confirm :) ). > Yes I did that and saw that. > > Really this just addressed one "hotspot" (decoding terms/postings from > the index); the query matching + scoring is also costly, and if you do > "other stuff" (highlighting, spell correction) that can be costly too > ... what kind of queries are you running / where are the hotspots in > profiling? > no 'other stuff' just matching and scoring -- of an embarrassingly complex query. I will post some results of profiling tomorrow. I had profiled extensively with lucene 3, we just got the code moved to lucene 4.3, and the very first thing I did was run this. In lucene 3 there was a very busy PriorityQueue in there somewhere; but I don't want to waste time and bandwidth on details until they are 4.x details. > > Mike McCandless > > http://blog.mikemccandless.com > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org > >
