Hi Chris, Thanks for your comment. You are correct, it looks to be CPU bound. However I am monitoring CPU utilization on all cores (40) and don't see close to 100% utilization on 20 cores, but all cores show activity ranging from %20 to 80% with lots of ups and downs. Can you help me understand all threads running during indexing? What is the best way to tune the number of threads (indexing and merging) based on the available hardware?
Thank you, Anahita -----Original Message----- From: Chris Hostetter [mailto:hossman_luc...@fucit.org] Sent: Thursday, April 30, 2015 11:32 AM To: java-user@lucene.apache.org Cc: Anahita Shayesteh-SSI Subject: Re: Lucene indexing speed on NVMe drive : Hi. I am studying Lucene performance and in particular how it benefits from faster I/O such as SSD and NVMe. : parameters as used in nightlyBench. (Hardware: Intel Xeon, 2.5GHz, 20 : processor ,40 with hyperthreading, 64G Memory) and study indexing speed ... : I get best performance (200GB/hour) with 20 indexing threads, increasing : number of threads to 40 hurts performance. Similarly increasing : maxConcurrentMerges above 3-5 doesn't seem to give me any benefit. I am : wondering what the bottleneck is, or anyone has insight on set of Maybe i'm missing something, but it sounds like you are CPU bound. Hyperthreading isn't going to help you if you are maxing out 20 (real) CPUS -- IIUC it only helps with some additional paralellization when processes are blocked by something else -- ie: IO bound. -Hoss http://www.lucidworks.com/ --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
