Hey Ryanne, thanks for the feedback. Would you mind going into more detail with how that would work?
My understanding is that the Kafka wire protocol isn’t really intended to be loadbalanced. Or is it more that the load balancer acts as more of a service discovery point for clients? I’ve seen in other threads that LinkedIn typically fronts their clusters with a load balancer, but I haven’t found much information on that strategy and the associated tradeoffs. Best, On Thu, Aug 29, 2019 at 12:36 AM Ryanne Dolan <ryannedo...@gmail.com> wrote: > Will and Garvit, you can use a load balancer with health checks for this > purpose. > > Ryanne > > On Wed, Aug 28, 2019, 6:09 PM Will Weber <rwawe...@gmail.com> wrote: > > > Apologies for piggybacking on a thread, figured the discussion was pretty > > relevant to a thought I had kicking around my brain. > > > > In the event of complete failure or sustained loss of connectivity of the > > first cluster, could the secondary cluster act as a failover for a given > > configuration? > > > > Assuming the respective clusters are set up like the following: > > > > Cluster A > > brokerA01:9092 > > brokerA02:9092 > > brokerA03:9092 > > > > Cluster B > > brokerB01:9092 > > brokerB02:9092 > > brokerB03:9092 > > > > And the corresponding producer's properties line contained something > like: > > > > > > > bootstrap.servers="brokerA01:9092,brokerA02:9092,brokerA03:9092,brokerB01:9092,brokerB02:9092,brokerB03:9092" > > > > My initial theory is that a failover action would respond like: > > 1. Attempting to reach brokerA01:9092 with a metadata fetch, cannot > reach, > > attempt to reach next broker > > 2. Attempting to reach brokerA02:9092 with a metadata fetch, cannot > reach, > > attempt to reach next broker > > 3. Attempting to reach brokerA03:9092 with a metadata fetch, cannot > reach, > > attempt to reach next broker > > 4. Attempting to reach brokerB01:9092 with a metadata fetch, able to > reach, > > begin exchange with reachable cluster > > > > I have a feeling that this would not work, but I don't really have any > > evidence to substantiate it at this point. > > > > Anyone have any experience on this sort of a situation? > > The intended goal is more to ensure that the messages end up on *some > > *cluster, not > > necessarily a particular cluster. > > > > Best, > > > > On Wed, Aug 21, 2019 at 3:28 AM David Jacot <dja...@confluent.io> wrote: > > > > > Hello, > > > > > > As of today, the producer is able to talk to only one Kafka cluster. > > > *bootstrap.servers* is to provide a list of brokers belonging to the > same > > > cluster to ensure at least one is available. > > > > > > In your case, you have to replicate the data from the first cluster to > > the > > > second cluster with mirror maker for instance. > > > > > > Best, > > > David > > > > > > On Mon, Aug 12, 2019 at 11:13 AM lsroudi abdel <lsro...@gmail.com> > > wrote: > > > > > > > I guess it more about replication, you can push your data in one > > choosen > > > > cluster and replicate your data to the second one by using mirror > maker > > > or > > > > confluent replicator or the new open sourced project by LinkedIn, > > > > "Brooklin" > > > > > > > > Le lun. 12 août 2019 à 10:19, Garvit Sharma <eng.gar...@gmail.com> a > > > > écrit : > > > > > > > > > Thanks, Isroudi. But in my usecase there are two separate zookeeper > > as > > > > > well. Let me know how would it handle. > > > > > > > > > > On Mon, Aug 12, 2019 at 1:45 PM lsroudi abdel <lsro...@gmail.com> > > > wrote: > > > > > > > > > > > If it is two cluster it mean you have two zookeeper, I guess you > > > can't > > > > do > > > > > > that, I f you have one zookeeper it will be ok, in the case you > > have > > > > one > > > > > > zookeeper it's just an architecture with replica across two data > > > > center, > > > > > I > > > > > > hope it clear for you > > > > > > > > > > > > Le lun. 12 août 2019 à 09:22, Garvit Sharma < > eng.gar...@gmail.com> > > a > > > > > > écrit : > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > I have 2 kafka clusters in different data-centers so can I > > provide > > > > the > > > > > > dns > > > > > > > hostnames of both the clusters separated by comma in > > > > > *bootstrap.servers* > > > > > > > key in producer config ? > > > > > > > > > > > > > > If I provide two different clusters in *bootstrap.servers *then > > how > > > > > would > > > > > > > the events get published ? > > > > > > > Would events get published to both the clusters or either of > them > > > > > > randomly > > > > > > > ? > > > > > > > > > > > > > > Looking forward to hearing from you. > > > > > > > > > > > > > > Thanks > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- Sent from my mobile device
