Will, the idea is to have a DNS name or VIP which resolves to IP addresses from one cluster or the other depending on health checks. You can have your producers load-balanced 50/50 between two clusters, or could have a backup cluster that is only used when the primary cluster is unreachable, etc.
When a producer times-out trying to send to the current set of IP address, it will try to resolve the DNS name again and receive a new set of IP addresses. For this to work, you may need to tweak various producer properties, e.g. retries, timeouts, refresh intervals. > Or is it more that the load balancer acts as more of a service discovery point for clients? Yes, that's an apt description. Ryanne On Thu, Aug 29, 2019 at 7:47 AM Will Weber <rwawe...@gmail.com> wrote: > Hey Ryanne, thanks for the feedback. > > Would you mind going into more detail with how that would work? > > My understanding is that the Kafka wire protocol isn’t really intended to > be loadbalanced. Or is it more that the load balancer acts as more of a > service discovery point for clients? > > I’ve seen in other threads that LinkedIn typically fronts their clusters > with a load balancer, but I haven’t found much information on that strategy > and the associated tradeoffs. > > Best, > > On Thu, Aug 29, 2019 at 12:36 AM Ryanne Dolan <ryannedo...@gmail.com> > wrote: > > > Will and Garvit, you can use a load balancer with health checks for this > > purpose. > > > > Ryanne > > > > On Wed, Aug 28, 2019, 6:09 PM Will Weber <rwawe...@gmail.com> wrote: > > > > > Apologies for piggybacking on a thread, figured the discussion was > pretty > > > relevant to a thought I had kicking around my brain. > > > > > > In the event of complete failure or sustained loss of connectivity of > the > > > first cluster, could the secondary cluster act as a failover for a > given > > > configuration? > > > > > > Assuming the respective clusters are set up like the following: > > > > > > Cluster A > > > brokerA01:9092 > > > brokerA02:9092 > > > brokerA03:9092 > > > > > > Cluster B > > > brokerB01:9092 > > > brokerB02:9092 > > > brokerB03:9092 > > > > > > And the corresponding producer's properties line contained something > > like: > > > > > > > > > > > > bootstrap.servers="brokerA01:9092,brokerA02:9092,brokerA03:9092,brokerB01:9092,brokerB02:9092,brokerB03:9092" > > > > > > My initial theory is that a failover action would respond like: > > > 1. Attempting to reach brokerA01:9092 with a metadata fetch, cannot > > reach, > > > attempt to reach next broker > > > 2. Attempting to reach brokerA02:9092 with a metadata fetch, cannot > > reach, > > > attempt to reach next broker > > > 3. Attempting to reach brokerA03:9092 with a metadata fetch, cannot > > reach, > > > attempt to reach next broker > > > 4. Attempting to reach brokerB01:9092 with a metadata fetch, able to > > reach, > > > begin exchange with reachable cluster > > > > > > I have a feeling that this would not work, but I don't really have any > > > evidence to substantiate it at this point. > > > > > > Anyone have any experience on this sort of a situation? > > > The intended goal is more to ensure that the messages end up on *some > > > *cluster, not > > > necessarily a particular cluster. > > > > > > Best, > > > > > > On Wed, Aug 21, 2019 at 3:28 AM David Jacot <dja...@confluent.io> > wrote: > > > > > > > Hello, > > > > > > > > As of today, the producer is able to talk to only one Kafka cluster. > > > > *bootstrap.servers* is to provide a list of brokers belonging to the > > same > > > > cluster to ensure at least one is available. > > > > > > > > In your case, you have to replicate the data from the first cluster > to > > > the > > > > second cluster with mirror maker for instance. > > > > > > > > Best, > > > > David > > > > > > > > On Mon, Aug 12, 2019 at 11:13 AM lsroudi abdel <lsro...@gmail.com> > > > wrote: > > > > > > > > > I guess it more about replication, you can push your data in one > > > choosen > > > > > cluster and replicate your data to the second one by using mirror > > maker > > > > or > > > > > confluent replicator or the new open sourced project by LinkedIn, > > > > > "Brooklin" > > > > > > > > > > Le lun. 12 août 2019 à 10:19, Garvit Sharma <eng.gar...@gmail.com> > a > > > > > écrit : > > > > > > > > > > > Thanks, Isroudi. But in my usecase there are two separate > zookeeper > > > as > > > > > > well. Let me know how would it handle. > > > > > > > > > > > > On Mon, Aug 12, 2019 at 1:45 PM lsroudi abdel <lsro...@gmail.com > > > > > > wrote: > > > > > > > > > > > > > If it is two cluster it mean you have two zookeeper, I guess > you > > > > can't > > > > > do > > > > > > > that, I f you have one zookeeper it will be ok, in the case you > > > have > > > > > one > > > > > > > zookeeper it's just an architecture with replica across two > data > > > > > center, > > > > > > I > > > > > > > hope it clear for you > > > > > > > > > > > > > > Le lun. 12 août 2019 à 09:22, Garvit Sharma < > > eng.gar...@gmail.com> > > > a > > > > > > > écrit : > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > I have 2 kafka clusters in different data-centers so can I > > > provide > > > > > the > > > > > > > dns > > > > > > > > hostnames of both the clusters separated by comma in > > > > > > *bootstrap.servers* > > > > > > > > key in producer config ? > > > > > > > > > > > > > > > > If I provide two different clusters in *bootstrap.servers > *then > > > how > > > > > > would > > > > > > > > the events get published ? > > > > > > > > Would events get published to both the clusters or either of > > them > > > > > > > randomly > > > > > > > > ? > > > > > > > > > > > > > > > > Looking forward to hearing from you. > > > > > > > > > > > > > > > > Thanks > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > Sent from my mobile device >
