Thank you all for the great pointers. It was monitoring dashboard graph that mislead to thinking that nodes were problematic. It is not nodes but high frequency terms queries. Queries that match 50% docs in the index have high latency of 1-2 seconds.
On Sat, Sep 30, 2023 at 7:13 AM Deepak Goel <deic...@gmail.com> wrote: > What are the kind of queries you have? Are they the same 80% of the time? > If yes, they will hit a few nodes all the time. > > On Sat, 30 Sept 2023, 15:07 Vincenzo D'Amore, <v.dam...@gmail.com> wrote: > > > Hi Rajani, > > > > how many documents do you have in your collection? > > The nodes that consistently display high latency, do they have the same > > replicas? I mean, are they the only nodes where such high latency queries > > can be executed? > > > > On Fri, Sep 29, 2023 at 9:29 PM rajani m <rajinima...@gmail.com> wrote: > > > > > Hi Solr Users, > > > > > > What could cause some nodes in a cluster to have high query > latency > > > when compared to the rest? > > > > > > It is Solr 9.1.1 cluster of 32 nodes. Cluster has one collection > divided > > > into 32 shards and replication 3, so each node has one Solr server > > hosting > > > 3 replicas. It is composite router collection so documents are evenly > > > distributed across collection. Node roles are default so all the nodes > > are > > > coordinator nodes and data nodes. The leader nodes are spread across > the > > > nodes. > > > > > > Should every shard not work the same and have close to same latency? > > Looks > > > like 3 of 32 nodes consistently display high latency ( 1-2 seconds > QTime) > > > compared to the rest of the nodes(~30 ms). What can cause this > behavior? > > > > > > > > > -- > > Vincenzo D'Amore > > >
