Hi Benson, On Mon, Sep 30, 2013 at 5:21 PM, Benson Margulies <ben...@basistech.com> wrote: > The multithreaded index searcher fans out across segments. How aggressively > does 'optimize' reduce the number of segments? If the segment count goes > way down, is there some other way to exploit multiple cores?
forceMerge[1], formerly known as optimize, takes a parameter to configure how many segments should remain in the index. Regarding multi-core usage, if your query load is high enough to use all you CPUs (there are alwas #cores queries running in parrallel), there is generally no need to use the multi-threaded IndexSearcher. The multi-threaded index searcher can however help in case all CPU power is not in use or if you care more about latency than throughput. It indeed leverages the fact that the index is splitted into segments to parallelize query execution, so a fully merged index will actually run the query in a single thread in any case. There is no way to make query execution efficiently use several cores on a single-segment index so if you really want to parallelize query execution, you will have to shard the index to do at the index level what the multi-threaded IndexSearcher does at the segment level. Side notes: - A single segment index only runs more efficiently queries which are terms-dictionary-intensive, it is generally discouraged to run forceMerge on an index unless this index is read-only. - The multi-threaded index searcher only parallelizes query execution in certain cases. In particular, it never parallelizes execution when the method takes a collector. This means that if you want to use TotalHitCountCollector to count matches, you will have to do the parallelization by yourself. [1] http://lucene.apache.org/core/4_4_0/core/org/apache/lucene/index/IndexWriter.html#forceMerge%28int%29 -- Adrien --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
