Greetings Lucene Users
As a follow-up to my earlier mail:
We are also using Lucene segment warmers, as per recommendation,
segments per tier is now set to five, buffer memory is set to
(Runtime.getRuntime().totalMemory()*.08)/1024/1024;
See below for code used to instantiate writer:
LimitTokenCountAnalyzer limitAnalyzer = new
LimitTokenCountAnalyzer(application.getAnalyzerFactory().getAnalyzer(language,
AnalyzerFactory.Operation.INDEX), maxPerFieldTokens);
IndexWriterConfig conf = new
IndexWriterConfig(Version.LUCENE_46, limitAnalyzer);
TieredMergePolicy logMergePolicy = new
TieredMergePolicy();
logMergePolicy.setSegmentsPerTier(5);
conf.setMergePolicy(logMergePolicy);
conf.setRAMBufferSizeMB(bufferMemoryMB);
writer = new IndexWriter(fsDirectory, conf);
writer.getConfig().setMergedSegmentWarmer(readerWarmer);
This particular monster 24 core machine has 110G of RAM. I suppose one
possibly is to load the indexes that aren' t being changed into RAM on
startup. However, the indexes are already residing on fast SSD drives.
We're using the following JRE parameters:
-XX:+UseG1GC -XX:MaxGCPauseMillis=100 -XX:SurvivorRatio=3
-XX:+AggressiveOpts.
Let me know if there is anything else, we can try to obtain performance
gains.
Much appreciate
Jamie
On 2014/06/20, 9:51 AM, Jamie wrote:
Hi All
Thank you for all your suggestions. Some of the recommendations hadn't
yet been implemented, as our code base was using older versions of
Lucene with reduced capabilities. Thus, far, all the recommendations
for fast search have been implemented (e.g. using pagination with
searchAfter, DirectoryReader.openIfChanged, avoiding wrapping lucene
scoreDoc results, option to disable sorting, etc.).
While, in some environments, search performance has improved
significantly, in other larger ones we are unfortunately, still seeing
1 minute - 5 minute search times. For instance, in one site, the total
index size is 500GB with 190 million documents indexed. They are
running a machine with 24 core and 4 SSD drives to house the indexes.
New emails are being added to the indexes at a rate of 10 message/sec.
One area possible area for improvement: Searching is being conducted
across several indexes. To accomplish this, on each search, a
MultiReader is constructed, that consists of several subreaders
created by the DirectoryReader.openIfChangedMethod. Only one of the
indexes is updated frequently, the others are never updated. For each
search, a new IndexSearcher is created passed the MultiReader in the
constructor. From what I've read, MultiReader and IndexSearcher are
relatively lightweight and should not impact search performance. Is
this correct? Is there a faster way to handle searching across
multiple indexes? What is the performance impact of searching across
multiple indexes?
Am I correct that using SearchManager can't be used with a MultiReader
and NRT? I would appreciate all suggestions on how to optimize our
search performance further. Search time has become a usability issue.
Much appreciate
Jamie
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org
