> And why is per shard latency so high? (~120 ms)
ok. this proves that it's a problem with low level search.
Make sure you have swap disabled, and plenty of ram available for mmaping
(limited heap).
Then, you need to put a search load: some simple script looping keywords
with cache and distrib disabled. Check that under such load latency is
still higher. Then, you need to employ some sampling profiler. Usually just
looking into java stacktraces or solr admin/threads is enough. It gives you
a hotspot: performance bottleneck. Also it's worth to check io performance:
copy large file into /dev/nul and check read throughput. Then, check iostat
under search load. Compare iostat numbers with the reference numbers (I
don't have ones in my hands now).
On Sat, Nov 18, 2023 at 6:10 PM rajani m <rajinima...@gmail.com> wrote:
> q=cats&defType=edismax&qf=keywords&fl=id&rows=10
> &distrib=false&cache=false&debug=timing
>
> For query with distrib=false and cache=false, most shards have Qtime ~120
> and few of them have QTime 0. It is the same with cache=true or without
> cache param in the query.
> debug time response from one shard is as following - there is only single
> entry under process time and it is query time same as QTime
> "process": {"time": 107,"query": {"time": 107
>
>
>
> Same query with distrib=true, QTime avg is ~315. Is this expected Qtime for
> an aggregated query? And why is per shard latency so high? (~120 ms)
>
>
> On Fri, Nov 17, 2023 at 4:31 PM Mike Drob <md...@mdrob.com> wrote:
>
> > Maybe also experiment with cache=false
> >
> > On Fri, Nov 17, 2023 at 2:38 PM Mikhail Khludnev <m...@apache.org>
> wrote:
> >
> > > > What causes this issue?
> > > You may try to find an answer with distrib=false and debug=timing
> > >
> > >
> > > On Fri, Nov 17, 2023 at 8:49 PM rajani m <rajinima...@gmail.com>
> wrote:
> > >
> > > > Hi again,
> > > >
> > > > Thank you for all the pointers, they were very helpful. After
> > digging
> > > in
> > > > enough, I figured that it is a certain text field that matches a
> large
> > > set
> > > > of docs for a given query. And it is the one adding to the latency.
> > > > Appreciate any suggestions to optimize it.
> > > >
> > > > An example query that matches a large set of documents.
> > > >
> > > > q=cats&defType=edismax&qf=keywords description title&fl=id&rows=10 |
> > > Qtime
> > > > = 531
> > > >
> > > > The query fields are the same field type(text_en). I tried the
> > following
> > > > variants and have included their respective QTime notes.
> > > > 1. rows=0 | Qtime = 518
> > > > 2. qf=keywords | Qtime = 323 - This is the field that matches 85% of
> > > total
> > > > docs and is adding to the overall latency of the queries
> > > > 3. qf=description | Qtime= 144 - This field matches 25% of total docs
> > > > 4. qf=title | Qtime = 17 - This field matches 5% of total docs
> > > >
> > > > It is clear that whenever a query matches a large number of docs from
> > any
> > > > query fields, which is often the keywords field in this case, it has
> > high
> > > > latency. What causes this issue? What are my options to optimize
> such
> > > > queries latency?
> > > >
> > > > Thank you,
> > > > Rajani
> > > >
> > > > On Thu, Nov 2, 2023 at 12:30 PM Mikhail Khludnev <m...@apache.org>
> > > wrote:
> > > >
> > > > > On Thu, Nov 2, 2023 at 5:01 AM rajani m <rajinima...@gmail.com>
> > wrote:
> > > > >
> > > > > > Sorry, it took too long to get back to this one.
> > > > > >
> > > > > > The search query "http://host:8983/solr/v9/select?&q=*&rows=10";
> > > > > > consistently
> > > > > > took ~500 ms. With "distrib=false" all the 96 shards have QTime
> > 0-25
> > > > ms.
> > > > > > Does this mean aggregation of results from all the shards is
> taking
> > > > ~475
> > > > > > ms? I also tried shards.rows=5 and it still returned in ~475 ms
> > query
> > > > > time.
> > > > > >
> > > > >
> > > > > Giving that. Don't you have any limits for ShardHandlerFactory set
> in
> > > > > solr.xml?
> > > > > debugQuery=true or debug=track can produce verbose shard request
> > > tracing
> > > > > info. Perhaps it might get some clue.
> > > > > Full fledged approach is
> > > > >
> > > > >
> > > >
> > >
> >
> https://solr.apache.org/guide/solr/latest/deployment-guide/distributed-tracing.html
> > > > > for sure.
> > > > > Might it be a problem of slow dns names resolution?
> > > > > Does it still slow on rows=0?
> > > > > --
> > > > > Sincerely yours
> > > > > Mikhail Khludnev
> > > > >
> > > >
> > >
> > >
> > > --
> > > Sincerely yours
> > > Mikhail Khludnev
> > >
> >
>
--
Sincerely yours
Mikhail Khludnev
