Did you yet dig through the mailing list thread link that I posted earlier in this thread? It explains in more depth, suggests a number of possible mitigations, and has a bunch of links to jira issues that provide extra context. Off the cuff, I'd say that setting `enableGraphQueries=false` may be most immediately helpful in terms of restoring performance.
(As an aside: from my perspective though, even if you can restore performance, it would be at the expense of nuances of functionality. Longer term I'd really like to help solve this properly, involving some combination of the issues linked to in the above thread ...) Michael On Thu, May 5, 2022 at 3:01 AM Satya Nand <satya.n...@indiamart.com.invalid> wrote: > Hi Michael, > > > 1. set `pf=` (phrase field empty string), disabling implicit phrase query > > building. This would help give a sense of whether phrase queries are > > involved in the performance issues you're seeing. > > > We are also in the process of moving from standalone 6.6 to 8.7 Solr cloud, > We also noticed a huge response time increase (91 ms To 170 ms +). > > We tried applying the tweak of disabling the pf field and response time was > back to normal. So somehow pf was responsible for increased response time. > > We are using both query-time multi-term synonyms and > WordDelimiter[Graph]Filter. > > What should we do next from here as we can't disable the pf field? > > Cluster Configuration: > > 3 Solr Nodes: 5 CPU, 42 GB Ram (Each) > 3 Zookeeper Nodes: 1 CPU, 2 GB Ram (Each) > 3 Shards: 42m Documents, 42 GB (Each) > Heap: 8 GB > > > There are no deleted documents in the cluster and no updates going on. We > are trying to match the performance first. > > > > > > > > On Sat, Mar 26, 2022 at 9:42 PM Michael Gibney <mich...@michaelgibney.net> > wrote: > > > Are you using query-time multi-term synonyms or > WordDelimiter[Graph]Filter? > > -- these can trigger "graph phrase" queries, which are handled _quite_ > > differently in Solr 8.11 vs 6.5 (and although unlikely to directly cause > > the performance issues you're observing, might well explain the > performance > > discrepancy). If you're _not_ using either of those, then the rest of > this > > message is likely irrelevant. > > > > One thing to possibly keep an eye out for (in addition to gathering more > > evidence, as Mike Drob suggests): 6.5 started using span queries for > "graph > > phrase" queries (LUCENE-7699), but the resulting phrase queries were > > completely ignored in Solr (bug) until 7.6 (SOLR-12243). Completely > > ignoring complex phrase queries did however greatly reduce latency and > CPU > > load on 6.5! > > > > 7.6 started paying attention to these queries again (SOLR-12243), but > also > > went back to "fully-enumerated" combinatoric approach to phrase queries > > when `ps` (phrase slop) is greater than 0 (LUCENE-8531). > > > > Some parameters you could tweak, assuming you're using edismax: > > 1. set `pf=` (phrase field empty string), disabling implicit phrase query > > building. This would help give a sense of whether phrase queries are > > involved in the performance issues you're seeing. > > 2. set `ps=0` (phrase slop 0), this should allow span queries to be > built, > > which should generally be more efficient than analogous non-span-query > > approach (basically this would make the change introduced by LUCENE-8531 > > irrelevant); tangentially: the special case building span queries for > > `ps=0` is removed as of Lucene 9.0 (will be removed as of Solr 9.0 -- not > > directly relevant to this issue though). > > > > Michael > > > > On Sat, Mar 26, 2022 at 8:26 AM Mike Drob <md...@mdrob.com> wrote: > > > > > Can you provide more details on what they CPU time is spent on? Maybe > > look > > > at some JFR profiles or collect several jstacks to see where they > > > bottlenecks are. > > > > > > On Sat, Mar 26, 2022 at 3:49 AM Modassar Ather <modather1...@gmail.com > > > > > wrote: > > > > > > > Hi, > > > > > > > > We are trying to migrate to Solr-8.11.0 from Solr-6.5.1. Following > are > > > the > > > > details of Solr installation. > > > > > > > > Server : EC2 instance with 32 CPUs and 521 GB > > > > Storage : EBS Volume. General Purpose SSD (gp3) with 3000/5000 IOPS > > > > Ubuntu :Ubuntu 20.04.3 LTS > > > > Java : openjdk 11.0.14 > > > > SolrCloud : 12 shards having a total 4+ TB index. Each node has a > 30GB > > > max > > > > memory limit. > > > > GC setting on Solr : G1GC > > > > Solr query timeout : 5 minutes > > > > > > > > During testing we observed a high CPU utilisation and few of the > > queries > > > > with wildcard queries are timing out. These queries are getting > > executed > > > > completely on Solr-6.5.1. > > > > After tuning a few of the parameters of GC settings the CPU > utilisation > > > > came down but it is still high when compared with Solr-6.5.1 and some > > > > queries with wildcard queries are still failing. > > > > > > > > Kindly provide your suggestions. > > > > > > > > Thanks, > > > > Modassar > > > > > > > > > >
