> no -- this won't work, because the requerst your remote client sends will > need to specify the actual collection you want to query, and when the node
I was more thinking of some explicit &collections=otherColl or &shards=other_1,other_2 but easier to just send to a node without that collection - we have 2 more nodes in the cluster > that seems... dangerous. you could easily wind up in a situation where > nodes just keep trying to forward forever? There is some special http parameter being added when forwarding requests, so I'm sure each node will be able to decide whether it should act as LB or if it is supposed to be the final destination. Or we can add such a param. Of course, if SolrJ on the client side has already selected a replica, the receiving node should not discard that and do its own balancing. So there is some state to get right here. Jan > 10. mar. 2021 kl. 19:32 skrev Chris Hostetter <hossman_luc...@fucit.org>: > > > : Is there any way whatsoever to solve this on the Solr side only? > : > : Only I can think of is to send all requests to a 3rd node in the cluster > : that does not have a core for the collection, then it will balance > : between the two :) > > correct -- you can create a Solr node w/o any cores that will act as a > "load balancer" to other solr nodes. > > : Or create a new, empty collection on the node, which acts as a routing > : collection only to the target collection? > > no -- this won't work, because the requerst your remote client sends will > need to specify the actual collection you want to query, and when the node > gets this it will hand it to the local core for that collection -- it > won't care that there is another local collection that's unrelated. > > : Sounds like there should be a way to explicitly disable the > : "optimization" of always handling the request locally in single-shard > : collections, i.e. always try to balance unless shards.preference=local? > > that seems... dangerous. you could easily wind up in a situation where > nodes just keep trying to forward forever? > > > > : > : Jan > : > : > 10. mar. 2021 kl. 19:06 skrev Chris Hostetter <hossman_luc...@fucit.org > <mailto:hossman_luc...@fucit.org>>: > : > > : > > : > : Ah, I missed "single shard" ... this looks relevant: > : > : https://issues.apache.org/jira/browse/SOLR-12217 > <https://issues.apache.org/jira/browse/SOLR-12217> > : > > : > That improvement still isn't going to impact Jan's situation where the > : > *client* isn't SolrJ ... as the description says: > : > > : >>> NOTE: This Jira doesn't cover the single-sharded collections cases when > : >>> not using the CloudSolrClient or Streaming Expressions (i.e. if you do > : >>> a non-streaming curl request to a random node in the cluster, the > : >>> shards.preference parameter is not considered in the case of single > : >>> shards collections). > : > > : > > : > : > : > : On Wed, Mar 10, 2021 at 12:43 PM Jan Høydahl <jan....@cominvent.com > <mailto:jan....@cominvent.com>> wrote: > : > : > : > : > We have not set any shard.preference, and I also think preferLocal > : > : > defaults to false, i.e random > : > : > > : > : > Earlier we had 2 shares for the same collection (both existed on both > : > : > nodes) and then requests were distributed to both nodes. That’s why, > when > : > : > we went to 1 shard, I was wondering if the “single-shard” code path > perhaps > : > : > never attempts to utilize replicas?? But have not looked in code yet. > : > : > > : > : > Guess next step is to setup a small local test cluster and see what > : > : > happens. > : > : > > : > : > Jan Høydahl > : > : > > : > : > > 10. mar. 2021 kl. 15:46 skrev Michael Gibney > <mich...@michaelgibney.net <mailto:mich...@michaelgibney.net> > : > : > >: > : > : > > > : > : > > You say not "anything fancy" -- depending on how you define > "fancy", if > : > : > you > : > : > > have an explicit `shards.preference` param, based on the version > you're > : > : > > running (8.4) you might also take a look at > : > : > > https://issues.apache.org/jira/browse/SOLR-14471 > <https://issues.apache.org/jira/browse/SOLR-14471>. (If SOLR-14471 is the > : > : > > problem, removing the explicit `shards.preference` param should > restore > : > : > > default "shuffling" routing). > : > : > > > : > : > > I haven't dug too deep, but it looks like for 8.4 preferLocalShards > : > : > > actually defaults to false? I might be missing something though: > : > : > > > : > : > > https://github.com/apache/lucene-solr/blob/releases/lucene-solr/8.4.1/solr/solrj/src/java/org/apache/solr/client/solrj/routing/RequestReplicaListTransformerGenerator.java#L85 > > <https://github.com/apache/lucene-solr/blob/releases/lucene-solr/8.4.1/solr/solrj/src/java/org/apache/solr/client/solrj/routing/RequestReplicaListTransformerGenerator.java#L85> > : > : > > > : > : > > > : > : > > > : > : > >> On Wed, Mar 10, 2021 at 9:10 AM Houston Putman > <houstonput...@gmail.com > : > : > > > : > : > >> wrote: > : > : > >> > : > : > >> I could be wrong, but i dont think preferLocalShards is the > default in > : > : > >> multi-shard use cases. > : > : > >> > : > : > >>> On Wed, Mar 10, 2021 at 9:07 AM Mike Drob <md...@mdrob.com> wrote: > : > : > >>> > : > : > >>> I believe a server will always try to prefer local cores. Can you > do an > : > : > >>> experiment with 3 nodes, and send http queries to the node not > hosting > : > : > >> any > : > : > >>> replicas? That should confirm the balanced distribution. > : > : > >>> > : > : > >>> If you have multiple shards, the receiving server will forward the > : > : > >> requests > : > : > >>> for shards it doesn’t have, but would still prefer local shards > when > : > : > they > : > : > >>> are available. > : > : > >>> > : > : > >>> On Wed, Mar 10, 2021 at 8:00 AM Jan Høydahl > <jan....@cominvent.com> > : > : > >> wrote: > : > : > >>> > : > : > >>>> Hi, > : > : > >>>> > : > : > >>>> A client has a SolrCloud 8.4 setup with two nodes, and one > collection > : > : > >>> with > : > : > >>>> one shard and replicationFactor=2. > : > : > >>>> Of course we want search traffic to be evenly distributed > between the > : > : > >> two > : > : > >>>> replicas. > : > : > >>>> The client is using plain HTTP requests, no SolrJ or anything > fancy, > : > : > >> and > : > : > >>>> sends all requests to one of the two nodes. > : > : > >>>> I was expecting Solr to forward about 50% of those requests to > the > : > : > >> other > : > : > >>>> replica, but it is serving them all locally. > : > : > >>>> > : > : > >>>> I know we can setup an LB in front or re-program the client to do > : > : > round > : > : > >>>> robin, but that is not my question. > : > : > >>>> Is the select-random-replica logic only active when we have a > sharded > : > : > >>>> oollection, and not for a single-shard? > : > : > >>>> > : > : > >>>> Jan > : > : > >>> > : > : > >> > : > : > > : > : > : > > : > -Hoss > : > http://www.lucidworks.com/ <http://www.lucidworks.com/> > : > : > > -Hoss > http://www.lucidworks.com/ <http://www.lucidworks.com/>
