Re: Suggestions to improve Star queries latencies

On 4/16/23 07:24, Rajani Maski wrote: What are your suggestions to improve star queries latencies? By star queries I mean "*:*" or single term queries having boost formulas (such as doc recency and many others) taking 10 or more seconds. It is a large collection with good compute resources, howe

Re: Suggestions to improve Star queries latencies

> And why is per shard latency so high? (~120 ms) ok. this proves that it's a problem with low level search. Make sure you have swap disabled, and plenty of ram available for mmaping (limited heap). Then, you need to put a search load: some simple script looping keywords with cache and distrib di

Re: Suggestions to improve Star queries latencies

q=cats&defType=edismax&qf=keywords&fl=id&rows=10 &distrib=false&cache=false&debug=timing For query with distrib=false and cache=false, most shards have Qtime ~120 and few of them have QTime 0. It is the same with cache=true or without cache param in the query. debug time response from one shard is

Re: Suggestions to improve Star queries latencies

Maybe also experiment with cache=false On Fri, Nov 17, 2023 at 2:38 PM Mikhail Khludnev wrote: > > What causes this issue? > You may try to find an answer with distrib=false and debug=timing > > > On Fri, Nov 17, 2023 at 8:49 PM rajani m wrote: > > > Hi again, > > > > Thank you for all the

Re: Suggestions to improve Star queries latencies

> What causes this issue? You may try to find an answer with distrib=false and debug=timing On Fri, Nov 17, 2023 at 8:49 PM rajani m wrote: > Hi again, > > Thank you for all the pointers, they were very helpful. After digging in > enough, I figured that it is a certain text field that matc

Re: Suggestions to improve Star queries latencies

Hi again, Thank you for all the pointers, they were very helpful. After digging in enough, I figured that it is a certain text field that matches a large set of docs for a given query. And it is the one adding to the latency. Appreciate any suggestions to optimize it. An example query that mat

Re: Suggestions to improve Star queries latencies

On Thu, Nov 2, 2023 at 5:01 AM rajani m wrote: > Sorry, it took too long to get back to this one. > > The search query "http://host:8983/solr/v9/select?&q=*&rows=10"; > consistently > took ~500 ms. With "distrib=false" all the 96 shards have QTime 0-25 ms. > Does this mean aggregation of result

Re: Suggestions to improve Star queries latencies

On 11/1/23 20:01, rajani m wrote: The search query "http://host:8983/solr/v9/select?&q=*&rows=10"; consistently took ~500 ms. With "distrib=false" all the 96 shards have QTime 0-25 ms. Does this mean aggregation of results from all the shards is taking ~475 ms? I also tried shards.rows=5 and i

Re: Suggestions to improve Star queries latencies

edge n gram tokenizer could be useful . that would shift query time compute to index time at the cost of bigger index size. Sent from my iPhone > On Nov 1, 2023, at 7:02 PM, rajani m wrote: > > Sorry, it took too long to get back to this one. > > The search query "http://host:8983/solr/v9/s

Re: Suggestions to improve Star queries latencies

Sorry, it took too long to get back to this one. The search query "http://host:8983/solr/v9/select?&q=*&rows=10"; consistently took ~500 ms. With "distrib=false" all the 96 shards have QTime 0-25 ms. Does this mean aggregation of results from all the shards is taking ~475 ms? I also tried shards

Re: Suggestions to improve Star queries latencies

> It is a query with popularity and recency boosts, requesting the first 100 > docs with 3 fields per doc. It sounds like you are scoring/sorting, so the optimization that Mikhail mentioned would not apply (your use-case is not "sort-irrelevant"). Can you share more about specifically how your imp

Re: Suggestions to improve Star queries latencies

To send the query to a single shard you can add the parameter "distrib=false" to the query and it will stay on that shard. Joel Bernstein http://joelsolr.blogspot.com/ On Wed, Apr 19, 2023 at 5:21 PM Joel Bernstein wrote: > You're hunting for a bottleneck. Here is how I would go about finding

Re: Suggestions to improve Star queries latencies

You're hunting for a bottleneck. Here is how I would go about finding it: First I would run the query on a single shard and see how long it takes. If the single shard is slow you've found your bottleneck. If its fast then try the same query on each shard, one of the shards might be slow and you've

Re: Suggestions to improve Star queries latencies

Thank you, Mikhail. On Wed, Apr 19, 2023 at 7:59 AM Mikhail Khludnev wrote: > Hello, Rajani. > I meant [SOLR-14765] optimize DocList creation by skipping sort for > sort-irrelevant cases - ASF JIRA (apache.org) > > > On Wed, Apr 19, 2023 at 4:0

Re: Suggestions to improve Star queries latencies

Hello, Rajani. I meant [SOLR-14765] optimize DocList creation by skipping sort for sort-irrelevant cases - ASF JIRA (apache.org) On Wed, Apr 19, 2023 at 4:05 AM Rajani Maski wrote: > Hi Mikhail, > >Yes, 9.1.1, that should be helpful, can you

Re: Suggestions to improve Star queries latencies

Do you really have 96 separate disks and memory for each shard? They seemed a bit small and numerous to me, unless you are trying to fit every shard into memory of separate nodes and have the hardware resources for it — > On 19 Apr 2023, at 05:43, Rajani Maski wrote: > > It is a query with po

Re: Suggestions to improve Star queries latencies

It is a query with popularity and recency boosts, requesting the first 100 docs with 3 fields per doc. No facets. It is a query against a collection of 96 shards ~7m docs per shard. Could the cause for latency be boost queries and would it also be time spent in aggregating results from many shards

Re: Suggestions to improve Star queries latencies

I think there are more important questions here. What do you want with a *:* query? Do you want all the results in on return? Or do you just want the count of total documents? Or to put the results in facets? *:* should never take long unless you are requesting every single document not just

Re: Suggestions to improve Star queries latencies

Hi Mikhail, Yes, 9.1.1, that should be helpful, can you please point me to the related jira(s) and/or docs? Thank you, Rajani On Mon, Apr 17, 2023 at 2:09 AM Mikhail Khludnev wrote: > Hello Rajani. > Which version are you running? IIRC 9.1.2 has some > improvement about caching short quer

Re: Suggestions to improve Star queries latencies

Hello Rajani. Which version are you running? IIRC 9.1.2 has some improvement about caching short queries. On Sun, Apr 16, 2023 at 4:25 PM Rajani Maski wrote: > Hi Solr Users, > > What are your suggestions to improve star queries latencies? By star > queries I mean "*:*" or single term queries ha