Greetings Lucene Users

As a follow-up to my earlier mail:

We are also using Lucene segment warmers, as per recommendation, segments per tier is now set to five, buffer memory is set to (Runtime.getRuntime().totalMemory()*.08)/1024/1024;

See below for code used to instantiate writer:

LimitTokenCountAnalyzer limitAnalyzer = new LimitTokenCountAnalyzer(application.getAnalyzerFactory().getAnalyzer(language, AnalyzerFactory.Operation.INDEX), maxPerFieldTokens); IndexWriterConfig conf = new IndexWriterConfig(Version.LUCENE_46, limitAnalyzer); TieredMergePolicy logMergePolicy = new TieredMergePolicy();
                    logMergePolicy.setSegmentsPerTier(5);
                    conf.setMergePolicy(logMergePolicy);
                     conf.setRAMBufferSizeMB(bufferMemoryMB);
                    writer = new IndexWriter(fsDirectory, conf);
writer.getConfig().setMergedSegmentWarmer(readerWarmer);


This particular monster 24 core machine has 110G of RAM. I suppose one possibly is to load the indexes that aren' t being changed into RAM on startup. However, the indexes are already residing on fast SSD drives.

We're using the following JRE parameters:

-XX:+UseG1GC -XX:MaxGCPauseMillis=100 -XX:SurvivorRatio=3 -XX:+AggressiveOpts.

Let me know if there is anything else, we can try to obtain performance gains.

Much appreciate

Jamie
On 2014/06/20, 9:51 AM, Jamie wrote:
Hi All

Thank you for all your suggestions. Some of the recommendations hadn't yet been implemented, as our code base was using older versions of Lucene with reduced capabilities. Thus, far, all the recommendations for fast search have been implemented (e.g. using pagination with searchAfter, DirectoryReader.openIfChanged, avoiding wrapping lucene scoreDoc results, option to disable sorting, etc.).

While, in some environments, search performance has improved significantly, in other larger ones we are unfortunately, still seeing 1 minute - 5 minute search times. For instance, in one site, the total index size is 500GB with 190 million documents indexed. They are running a machine with 24 core and 4 SSD drives to house the indexes. New emails are being added to the indexes at a rate of 10 message/sec.

One area possible area for improvement: Searching is being conducted across several indexes. To accomplish this, on each search, a MultiReader is constructed, that consists of several subreaders created by the DirectoryReader.openIfChangedMethod. Only one of the indexes is updated frequently, the others are never updated. For each search, a new IndexSearcher is created passed the MultiReader in the constructor. From what I've read, MultiReader and IndexSearcher are relatively lightweight and should not impact search performance. Is this correct? Is there a faster way to handle searching across multiple indexes? What is the performance impact of searching across multiple indexes?

Am I correct that using SearchManager can't be used with a MultiReader and NRT? I would appreciate all suggestions on how to optimize our search performance further. Search time has become a usability issue.

Much appreciate

Jamie


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org

Reply via email to