Re: [Suggestions Required] 110 Concurrency users indexing on Lucene dont finish in 200 ms.

2014-02-11 Thread Michael McCandless
Lucene's indexing is very concurrent, especially in 4.x where we've added concurrent flushing: http://blog.mikemccandless.com/2011/05/265-indexing-speedup-with-lucenes.html Are you sure the bottleneck is in Lucene and not e.g. the server-side handling of suddenly receiving 110 X 10,50,100 document

Getting term ords during collect

2014-02-11 Thread Kyle Judson
Hi All, What are the ways I can get the ords for the terms of a particular field in the collect method of a Collector? I'm currently using a SortedSetDocValues that I obtained before the query but it's taking longer to load than I would like. Thanks Kyle

Re: Getting term ords during collect

2014-02-11 Thread Michael McCandless
SortedSetDV is probably the best way to do so. You could also encode the ords yourself into a byte[] and use binary DV. But why are you seeing it take too long to load? You can switch to different DV formats to tradeoff RAM usage and lookup speed.. Mike McCandless http://blog.mikemccandless.co

RE: Getting term ords during collect

2014-02-11 Thread Kyle Judson
Too long is always relative but one of the fields in a 24G index with 3.9M terms takes 2.5 min to load from SSD. I'm getting the SortedSetDocValues from FieldCache.DEFAULT.getDocTermOrds. What are the other DV formats? I'll look them up and try them. Thanks Kyle > From: luc...@mikemccandless.c