First look at the system metrics. Is it CPU bound or IO bound? Each request is single threaded, so a CPU bound system will have one core used at roughly 100% for that time. An IO bound system will not be using much CPU but will have threads in iowait and lots of disk reads.
After you know that, then you know what to work on. If it is IO bound, get enough RAM for the OS, JVM, and index files to all be in memory. If it is CPU bound, get a faster processor and work on the config to have the request do less work. Sharding can also help. I’m not a fan of always choosing 31 GB for the JVM. Allocate only as much as is needed. Java will use the whole heap whether it is needed or not. You might only need 8 GB. All of our clusters run with 16 GB. That includes some machines with 36 cores. wunder Walter Underwood wun...@wunderwood.org http://observer.wunderwood.org/ (my blog) > On Mar 18, 2022, at 10:22 AM, Dave <hastings.recurs...@gmail.com> wrote: > > I’ve found that each solr instance will take as many cores as it needs per > request. Your 2 sec response sounds like you just started the server and then > did that search. I never trust the first search as nothing has been put into > memory yet. I like to give my jvms 31 gb each and let Linux cache the rest of > the files as it sees fit, with swap turned completely off. Also *:* can be > heavier than you think if you have every field indexed since it’s like a > punch card like system where all the fields have to match. > >> On Mar 18, 2022, at 12:45 PM, Vincenzo D'Amore <v.dam...@gmail.com> wrote: >> >> Thanks for your support, just sharing what I found until now. >> >> I'm working with SolrCloud with a 2 node deployment. This deployment has >> many indexes but a main one 160GB index that has become very slow. >> Select *:* rows=1 take 2 seconds. >> SolrCloud instances are running in kubernetes and are deployed in a pod >> with 128GB RAM but only 16GB to JVM. >> Looking at Solr Documentation I've found nothing specific about what >> happens to the performance if the number of CPUs is not correctly detected. >> The only interesting page is the following and it seems to match with your >> suggestion. >> At the end of paragraph there is a not very clear reference about how the >> Concurrent Merge Scheduler behavior can be impacted by the number of >> detected CPUs. >> >>> Similarly, the system property lucene.cms.override_core_count can be set >> to the number of CPU cores to override the auto-detected processor count. >> >>> Talking Solr to Production > Dynamic Defaults for ConcurrentMergeScheduler >>> >> https://solr.apache.org/guide/8_3/taking-solr-to-production.html#dynamic-defaults-for-concurrentmergescheduler >> >> >> >>> On Thu, Mar 17, 2022 at 1:22 PM Thomas Matthijs <li...@selckin.be> wrote: >>> >>> I don't know how it affects solr, but if you're interested in java's >>> support to detect cgroup/container limits on cpu/memory etc, you can use >>> these links as starting points to investigate. >>> It affect some jvm configuration, like initial GC selection & settings >>> that can affect performance. >>> It was only backported to java 8 quite recently, so if you're still on >>> that might want to check if you're on the latest version. >>> >>> https://bugs.java.com/bugdatabase/view_bug.do?bug_id=JDK-8146115 >>> https://bugs.openjdk.java.net/browse/JDK-8264136 >>> >>> >>>> On Thu, Mar 17, 2022, at 01:11, Vincenzo D'Amore wrote: >>>> Hi Shawn, thanks for your help. >>>> >>>> Given that I’ll put the question in another way. >>>> If Java don’t correctly detect the number of CPU how the overall >>>> performance can be affected by this? >>>> >>>> Ciao, >>>> Vincenzo >>>> >>>> -- >>>> mobile: 3498513251 >>>> skype: free.dev >>>> >>>>> On 16 Mar 2022, at 18:56, Shawn Heisey <elyog...@elyograg.org> wrote: >>>>> >>>>> On 3/16/22 03:56, Vincenzo D'Amore wrote: >>>>>> just asking how can I rely on the number of processors the solr >>> dashboard >>>>>> shows. >>>>>> >>>>>> Just to give you a context, I have a 2 nodes solrcloud instance >>> running in >>>>>> kubernetes. >>>>>> Looking at solr dashboard (8.3.0) I see there is only 1 cpu available >>> per >>>>>> solr instance. >>>>>> but the Solr pods are deployed in two different kube nodes, and >>> entering >>>>>> the pod with the >>>>>> kubectl exec -ti solr-0 -- /bin/bash >>>>>> and running top I see there are 16 cores available for each solr >>> instance. >>>>> >>>>> The dashboard info comes from Java, and Java gets it from the OS. How >>> that works with containers is something I don't know much about. Here's >>> what Linux says about a server I have which has two six-core Intel CPUs >>> with hyperthreading. This is bare metal, not a VM or container: >>>>> >>>>> elyograg@smeagol:~$ grep processor /proc/cpuinfo >>>>> processor : 0 >>>>> processor : 1 >>>>> processor : 2 >>>>> processor : 3 >>>>> processor : 4 >>>>> processor : 5 >>>>> processor : 6 >>>>> processor : 7 >>>>> processor : 8 >>>>> processor : 9 >>>>> processor : 10 >>>>> processor : 11 >>>>> processor : 12 >>>>> processor : 13 >>>>> processor : 14 >>>>> processor : 15 >>>>> processor : 16 >>>>> processor : 17 >>>>> processor : 18 >>>>> processor : 19 >>>>> processor : 20 >>>>> processor : 21 >>>>> processor : 22 >>>>> processor : 23 >>>>> >>>>> If I start Solr on that server, the dashboard reports 24 processors. >>>>> >>>>> Thanks, >>>>> Shawn >>>>> >>> >> >> >> -- >> Vincenzo D'Amore
